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Abstract 

The decision regarding whether to maintain Federal Aviation Administration 
certification of military commercial derivative aircraft (MCDA) can be extremely 
difficult due to the subjective nature of some of the associated costs and benefits. The 
purpose of this research is to help decision makers understand the level of impact those 
costs and benefits have on their decision and how that impact may change over the 
lifecycle of the MCDA fleet. Data regarding the various criteria involved in such 
decisions were collected from subject matter experts using the Delphi technique. This 
information was then incorporated into a decision support tool that uses the principles of 
multi-attribute decision making. The tool helps to illustrate the relative importance of 
each of the decision criteria based upon specific points in a typical MCDA fleet lifecycle. 
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IMPACT OF DECISION CRITERIA ON FEDERAL AVIATION ADMINISTRATION 


CERTIFICATION OF MILITARY COMMERCIAL DERIVATIVE AIRCRAFT 


I. Introduction 

Overview 

This thesis helps identify criteria that U.S. Air Force (USAF) leaders and decision 
makers can use detennine whether maintaining Federal Aviation Administration (FAA) 
certification of military commercial derivative aircraft (MCDA) is beneficial when 
compared to maintaining only military certification. This chapter helps to lay the 
groundwork for understanding what FAA and military certifications are and how they 
apply to MCDA. First, background on FAA and military certification is provided, 
followed by current problems and concerns Air Force leaders are facing regarding 
MCDA certification. Next, the focus of the research behind these issues is presented, as 
well as an outline of the primary research objectives. Finally, assumptions and 
limitations to the research are identified, along with how they might impact findings and 
conclusions. 

Background 

Over the last century, the field of U.S. aviation has grown from a relatively small 
industry with only a few aircraft used primarily for observation and mail delivery, into a 
thriving commercial and military industry that employs literally thousands of aircraft that 
play vital roles in our national defense. At an early stage of this industry’s growth, it was 
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imperative that some form of regulation be implemented in the areas of aircraft 
production, maintenance, and operations in order to ensure that fundamental levels of 
quality, safety, and consistency were achieved. An industry as large as aviation typically 
needs some form of regulation or a set of standardized practices that can be agreed to and 
then enforced by a central body or government agency (“A Brief History,” 2011). With 
the right level of regulatory oversight, a strong foundation of quality and safety become 
just as important to an organization’s success as profits or capabilities. Most military and 
civilian leaders alike would agree that there is value in following a well-documented 
practice of certifying the quality and airworthiness of aircraft as well as adhering to 
firmly established and consistent maintenance practices. 

Established early-on in the aviation industry, the aircraft and maintenance 
certification process has evolved into something very complex and specialized. Whether 
used to certify aircraft airworthiness, maintenance procedures, or maintenance personnel, 
achieving certification (FAA or otherwise) often requires following a set of complicated 
and circuitous steps that can sometimes take a great deal of time and technical knowledge 
to navigate. Adding to this complexity are the many types of certifications and 
certification waivers, as well as the implications that the requirements of each brings 
about over the lifecycle of the aircraft or process being certified. In any case, deciding 
which option is best (or most cost effective) for an organization can be extremely 
challenging, and is certainly a decision that should never be made without a thorough 
understanding of all possibilities. To be sure, this is an area in which leadership 
decisions have a serious impact on operational budgets, not to mention the impact they 
have on organizational quality and safety levels. 
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Additional consideration must also be given to the rate at which the certification 
process changes in relation to the rest of the industry. For example, not only do our 
decision makers and program managers need to decide whether or not military 
airworthiness certification is sufficient for the aircraft they manage, but they must also be 
able to detennine if the certification process itself is adequate. In other words, while we 
struggle with the decision over which type of certification to pursue, the reality is that the 
certification process itself, in a very broad sense, may in fact be outdated and hindering 
the technical growth of the industry. This is an important area of discussion that goes 
beyond the scope of this paper, but was identified as a potential area for future research. 
Problem Statement 

Considering that the military and civilian aviation industries do not necessarily 
take the same paths regarding requirements and regulations, it is possible that some 
confusion may exist with regard to USAF and FA A policies. Examples of differences 
may include the types of forms used to track aircraft parts or the FAA’s use of 
Airworthiness Directives to pass along safety alert messages (FAA FAQ, 2011). This is 
not to say that both organizations are not equally concerned with mission 
accomplishment, only that sometimes the primary focus of the FAA is safety, while the 
primary focus of military organizations is operational effectiveness (Bongiovi, Rob, 
Darrell Holcomb, Milt Ross, Bill Stockman, & Tony Perfilio, 2009). Overlapping areas 
or gaps in policy can become grey areas where interpretations may vary, and this can 
make it difficult for industry and military decision makers to make fact-based decisions. 
Instead, decisions might be made based on current interpretation, which can be biased by 
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factors such as current budget constraints or manpower levels. In the worst case, the 
resulting decision may not be optimal because it was based on incomplete infonnation. 

As this problem relates to the decisions being made regarding aircraft 
certification, current acquisition and sustainment decision criteria are typically not 
standardized in a way that assists leaders in detennining whether or not, from a life cycle 
management perspective, USAF aircraft should be managed as FAA certified commercial 
derivatives. Such determinations would include consideration of the various FAA 
certifications of aircraft, repair stations, and personnel. One of the difficult decisions 
facing our program managers today is whether or not pushing aircraft and supporting 
processes to follow FAA certification requirements is cost effective (Stockman, William, 
Milt Ross, Robert Bongiovi, & Greg Sparks, 2011). Before any of those questions can be 
answered accurately, there must first be a very firm and clear understanding of exactly 
what the benefits of maintaining certification might be, as well as associated costs, and 
the origins of both. To help clarify these areas, the researcher uses the Delphi method 
with subject matter experts (SME) having leadership and decision making experience in 
fields pertaining to USAF aircraft acquisition and sustainment. The goal of the study is 
to reach a consensus of opinion regarding the various risks and benefits associated with 
utilization of MCDA and subsequent FAA certification of those aircraft. The Delphi 
method is discussed in greater detail in the methodology section of this paper. 
Additionally, results of the Delphi study were used to develop a simple decision support 
tool that utilizes some of the aspects of Multiple Attribute Decision Making (MADM). 
The support tool and MADM attributes are also discussed in later chapters. 
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Research Focus 


The research will focus on trying to clearly identify the benefits associated with 
maintaining FAA certification, and how those benefits can change over the life cycle of 
an aircraft fleet. This is not a simple question of financial costs, as many of the potential 
benefits are fairly subjective and ambiguous. In that regard, some subjective weighting 
of each benefit will have to be made because, while FAA aircraft certification is similar 
to its military counterpart, it can be difficult to make detailed comparisons between the 
two. This is not to say that there is a great deal of difference between the quality levels or 
thoroughness of repair occurring within the military and commercial aviation industry, 
but simply that there are some differences in policy and administration. 

As with any large-scale process of this type, there will almost certainly never be a 
one-size-fits-all decision tool that will detennine conclusively if maintaining FAA 
certification is right for every aircraft type. In fact, it is likely that there are several 
criteria that must be analyzed for each individual decision, and results will be subjective. 
However, with a more thorough understanding of these criteria, decision makers should 
be able to more accurately detennine if the potential benefits outweigh the costs 
associated with the certifications required within their specific areas. 

Research Objectives 

The objective of this research will be to help identify clear and understandable 
(though sometimes very general) criteria for determining if maintaining FAA certification 
on MCDA currently in use (or being considered for future use) is warranted based on 
potential benefits or efficiencies achieved by it. In other words, it is the goal of this 
research to provide decision makers some additional tools to help them determine if 
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maintaining an ongoing partnership with the FAA is in their best interest, and if so, for 
how long. As discussed later in this paper, maintaining FAA certification is not 
necessarily an all-or-nothing prospect. In fact, there are many decision variables 
associated with milestones that occur during an aircraft life cycle that may actually lead 
to a decision to only maintain FAA certification for a relatively short period of time. As 
mentioned earlier, the final decision will likely be based on some fairly subjective and 
situation-specific measures. 

In an effort to produce a tool or model generic enough to have far reaching 
applicability while still maintaining validity, several underlying factors had to be 
considered, such as spare parts availability, type of maintenance support, and 
participation in a common parts pool. Also, many of these factors must be adjusted based 
on whether the solution chosen involves temporary FAA certification or certification over 
the entire life cycle of the system. 

Summary 

It can be difficult for decision makers to fully understand the impact of all aspects 
of FAA certification, and just as difficult to determine if maintaining that certification is 
the right course of action. This chapter outlined some of the problems faced while 
making that decision as well as how the focus of this research is aimed at clarifying the 
decision criteria. Chapter Two will provide background on the certification process 
itself, as well as outline some of the potential costs and benefits associated with 
maintaining FAA certification. The research methodology is then described in Chapter 
Three. Specifically, the use of the Delphi technique and multi-attribute decision making 
principles are explained with regard to the gathering of data and decision support tool 
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construction. Results of these methods are outlined in Chapter Four, with conclusions 
and recommendations provided in Chapter Five. 
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II. Literature Review 


Overview 

Supporting inputs used in the decisions regarding the use and duration of FAA 
certification of MCDA come from many levels. The professional backgrounds of 
personnel involved in these decisions can vary greatly, so it is important for all decision 
makers, regardless of level, to have a thorough understanding of the costs and benefits 
associated with USAF use of commercial derivatives, as well as the various military and 
FAA certifications required. A common understanding of the benefits associated with 
FAA certification must also be achieved. 

Due to the wide range of knowledge and experience of the personnel involved in 
the decision making, this chapter begins with a brief background of MCDA, the reasons 
behind aircraft certification, as well as the various types of certification. Next, a 
discussion of some of the primary benefits associated with maintaining FAA certification 
is provided as a means of establishing a common understanding in this area. The method 
behind identifying some of these benefits was the Delphi technique, which will be 
discussed later in the chapter. Finally, the chapter concludes with an analysis of MADM 
principles and how they were used to prioritize the benefits and show how those benefits 
might change over the life cycle of an aircraft. 

Military Commercial Derivative Aircraft (MCDA) 

At a very basic level, a military commercial derivative aircraft is based upon an 
existing or planned commercial design, but has typically been modified from its original 
configuration. As defined by AFI 62-601, USAF Airworthiness (11 June 2010), a MCDA 
is “any fixed- or rotary-wing aircraft procured as a commercial, FAA type certified, off-the- 



shelf non-developmental item, and whose serial number is listed on an FAA Type Certificate 


Data Sheet.” Nearly all military aircraft have some level commercial variation, which can 
range from minor parts or components to complete aircraft systems. In fact, a report 
released from the Defense Science Board in 2009 outlined eight levels of commercial 
systems that are used by the military, as shown in Table 1. 


Table 1: Levels of Commercial Systems (“Buying Commercial,” 2009) 


Level 

Definitions of "Commercial Systems" 

1 

Buy it from a manufacturer—domestic or foreign—and use it as is 

2 

Buy it from a manufacturer and make minor modifications; i.e., 

"paint it green" 

3 

Buy it from a manufacturer and make significant modifications, 
i.e., adding armored doors, guns, military radio, or a ballistically- 

tolerant fuel system 

4 

Have a manufacturer make significant modifications before 

buying it 

5 

Have a manufacturer gut an existing product and replace most of it 
with other (military-specified) parts 

6 

Have a manufacturer modify a commercial prototype product to 
meet military requirements 

7 

Have a manufacturer assemble a collection of commercial and 
military components independently qualified on different systems 

8 

A product that does not yet exist, but requires commercial 
development and utilizes commercial plants or processes 


Some of the benefits associated with military use of commercial derivatives 
diminish as the level number outlined in Table 1 increases. For example, Levels 1 and 2 
represent military use of commercial-off-the-shelf (COTS) systems (with only minor 
modification), and as such, development and production costs would likely be spread 
over a large pool of customers, significantly reducing costs to the USAF (“Buying 
Commercial,” 2009). One factor that helps to detennine the amount of cost savings 
achieved in such a situation is the point of the production run at which the aircraft is 
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purchased, and this will be discussed in greater detail in Chapter Three. There are 
potentially many other cost saving opportunities such as taking advantage of existing 
FAA testing and certification. Additional sustaimnent related items such as utilization of 
existing infrastructure, facilities, resources, and best practices, which will also be 
discussed in later sections, can be leveraged as well. However, as program managers 
consider using commercial derivatives, they must keep in mind that the costs associated 
with even minor modifications are high, and can quickly overtake the initial savings 
achieved by utilizing COTS systems (“Buying Commercial,” 2009). One way to avoid 
this is to ensure that military requirements closely match commercial derivative design. 

If they do not, then program managers may be considering factors in the higher levels of 
Table 1, and perhaps commercial derivatives would not be the right course of action. 
Other costs that are often overlooked are those associated with complying with Federal 
Acquisition Regulations (FAR). These regulations are the specific sets of rules that 
govern the federal acquisition process (Acquisition Central, 2012). For example, for 
complex purchases such as those discussed here, FAR requirements can add substantial 
costs to the process in the form of regulation compliance and oversight, and in fact, can 
make government contracts unappealing to potentially competing firms. Smaller 
contractors that do not have experience dealing with government contracts can quickly 
become overburdened by FAR requirements (Edwards, 2003). 

The Origin of Aircraft Certification 

The idea of aircraft certification has its roots in the subsidization of commercial 
ainnail routes in 1925 with the passage of the Kelly Airmail Act. Privatization of airmail 
delivery furthered the growth of the private aviation business and highlighted the need to 
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establish regulations that would standardize aviation practices and procedures. 

Additional pressure to enact this type of regulation came from General William “Billy” 
Mitchell, who was a proponent of an independent air force and pressured the U.S. 
President at the time, Calvin Coolidge, to pass new aviation legislation. To that effort, 
President Coolidge established a board to determine the status of aviation in the U.S., and 
chose Dwight Morrow, a Wall Street Banker, to lead it. This board came to be kn own as 
the Morrow Board, and their initial recommendations included establishing a Bureau of 
Air Navigation in order to address the overall lack of aviation infrastructure in the U.S., 
(since the U.S. was lagging Europe in this area), as well as establishing the Department of 
National Defense. Additionally, the board made recommendations regarding 
standardizing procedures, which further supported the need for government legislation of 
the aviation industry. Based on the results of the Morrow Board, President Coolidge 
eventually signed into law the Air Commerce Act of 1926 (Downs, 2001). 

The Air Commerce Act charged the Secretary of Commerce with, among many 
other things, enforcing air traffic rules, licensing pilots, and certifying aircraft. Several 
years later, the establishment of the Bureau of Air Commerce in 1934 and the Civil 
Aeronautics Authority in 1938 eventually culminated into the creation of the Federal 
Aviation Agency with the Federal Aviation Act of 1958. These continuing changes to the 
legislation and those enforcing it were required due to the quickly expanding nature of 
the industry. The name change from the Federal Aviation Agency to the Federal 
Aviation Administration (as it is known today) was the result of the establishment of the 
Department of Transportation under the Johnson administration, which encompassed the 
new Federal Aviation Administration (Federal Aviation, 2010). 


11 



These newly emerging agencies and regulations became ever more important as 
WWII began and the government looked to a fairly young aviation industry as the source 
for thousands of safe and reliable aircraft to support the war effort, as well as a source for 
the extensive supply chains necessary to support them. Until then, post WWI military 
aircraft that had flooded the commercial market had not been maintained consistently, 
and most flying organizations depended on the original equipment manufacturer (OEM) 
or organic maintenance for repair actions. However, as early aircraft were not initially 
designed (or expected) to last a long time, they were actually considered somewhat 
disposable, sometimes only used for a decade or less before being replaced by new 
designs. Leading up to and during WWII, most aircraft repairs were accomplished 
locally, and those aircraft that were not repairable were cannibalized for spare parts. This 
was generally the case regardless of whether the aircraft were for military or commercial 
use (Stockman et ah, 2011). 

In 1944, near the end of WWII, the U.S. invited its allies to meet and share ideas 
regarding the future of international civil aviation. This meeting, held in Chicago, 
ultimately led to the creation of the International Civil Aviation Organization (ICAO) as 
well as the development of the concept of a standard certification of aircraft 
airworthiness. It was believed that regulating the aviation industry and establishing 
airworthiness certification requirements would help ensure aircraft would be produced in 
accordance with approved designs and that manufacturers and operators would perfonn 
the required level of maintenance and alterations in accordance with standard policy 
(ICAO, 1944: Annex 8). 
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Following WWII, special demands for military transport required aircraft with 
very large and spacious interiors that were easy to load and unload, with the additional 
capability of landing and taking off on dirt landing strips. Because of these special 
requirements, the military developed unique aircraft that were specific to their missions. 
Aircraft such as the C-130, C-141, and others were produced that were not designed for 
use in commercial markets. These unique aircraft required their own military-specific 
maintenance infrastructure, to include organic maintainers and depot facilities (Feltus, 
2010). These types of separate military requirements, coupled with the manufacture of 
specific aircraft to meet them contributed to the Air Force’s eventual decision to develop 
their own airworthiness certification process independent from the FAA. Although in 
some cases, USAF aircraft had already received prior FAA certification (Humerick & 
Minnich, 1992) 

MCDA Certification Basics 

The FAA Military Certification Office (MCO) is the point of contact for the 
anned services with regard to MCDA certification, providing technical support, 
certification, and airworthiness services for MCDA (U.S. DOT Type Certification, 2007). 
The MCO also assists with establishing unique policies and procedures that address the 
challenges surrounding FAA support of military applications. The extent of these 
services provided by the FAA is further outlined in a Memorandum of Agreement 
(MOA) between the FAA and the military services, signed in 2007. The MOA provides 
details regarding the scope of support the FAA will provide for MCDA activities, as well 
as reporting requirements and approval authorities. Under this MOA, the FAA had also 
certificated military-only repair stations that perfonned maintenance and inspection of 
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parts and components being used solely by the military (also known as Part 145 
certification, in reference to the part of the FAA regulation covering repair station 
certification requirements) (FAA Interagency MO A, 2007). However, a review of the 
MOA that was conducted in 2009 detennined that the term “technical assistance” 
contained in the MOA did not provide for Part 145 certification of military repair 
stations. As such, a follow on memo signed in 2010 stated that the FAA would no longer 
issue such certificates to Department of Defense (DOD) entities performing work solely 
on U.S. military products (FAA Memorandum, 2010). 

Types of Certification 

FAA Certifications 

The certifications that the FAA normally provides for typical commercial aircraft 
include Type Certificates, Airworthiness Certificates, and Production Certificates. These 
certifications, defined in Table 2, do not represent an all-inclusive list, but are the primary 
certifications used to ensure public-use aircraft are designed, manufactured, tested, 
maintained, and operated in accordance with FAA standards. 


Table 2: Typical FAA Certifications for Commercial Aircraft (FAA FAQ, 2011) 


Certification 

Description 

FAA Type Certificate 

FAA approval of the design of a specific aircraft (or 
aircraft engine, propeller, etc.). Authorizes 
manufacturer to make product in the specific way 
covered by the certificate. Major design changes 
require issuance of supplemental or amend type 
certificates. 

FAA Airworthiness Certificate 

Authorization from the FAA to operate an aircraft. 
This certification means the FAA considers the 
aircraft safe to fly. 

FAA Production Certificate 

Approval to duplicate products under an FAA- 
approved type design (type certificate or 
supplemental type certificate, for example). 
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Military aircraft are considered public aircraft, and as such are not subject to FAR 
and FAA oversight. However FAA Order 8110.101 and the 2007 MOA with the military 
departments specifically outline FAA support for MCDA. Although, in order for the 
FAA to remain involved, there must be a similar civilian fleet and the DOD must ensure 
the aircraft is operated and maintained according to FAA standards. If not, the MCDA 
will not retain its FAA Type Certification, as was the case with the C-130J (Stockman et 
al„ 2011). 

Recently, the Secretary of the Air Force implemented Air Force Policy Directive 

(AFPD) 62-6, “USAF Airworthiness”, which outlines use of FAA certification with 

regard to commercial derivatives. The policy directive states, 

When a military mission is compatible with a certified civil usage, the AF will 
utilize FAA type certified commercial derivative aircraft (CDA) to the maximum 
extent practical. Commercial derivative aircraft whose primary mission is the 
transport of passengers shall be FAA type certified; FAA certification of these 
commercial derivative passenger carrying aircraft shall be maintained for the life 
of the air system. (AFPD 62-6, 2010:3) 

While certification decisions will be made on a case-by-case basis, this guidance 
certainly supports FAA certification as part of DOD acquisition strategy and policy. 
Military Certification 

While the USAF can use FAA evaluations and inspections for their MCDA, any 
elements that do not meet FAA certification requirements will be certified via approved 
military airworthiness requirements. These requirements are outlined in MIL-HDBK- 
516B, which states, 

Commercial derivative aircraft (CDA) are initially approved for safety of flight by 
the Federal Aviation Administration (FAA) and may have an FAA approved 
Certificate of Airworthiness. Any non-FAA approved alteration to a CDA may 
render all FAA certifications invalid. While alterations to CDA are covered by 
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rules unique to each branch of service, the operating service always has the 
responsibility for the airworthiness certification approval under public aircraft 
rules. (MIL-HDBK-516B, 2005:2-3) 

In essence, regardless whether FAA certification is maintained or not, the policy states 
that the underlying responsibility for airworthiness certification compliance lies with the 
applicable service branch. 

If FAA certification is not maintained, or for military certification of items not 
certified by the FAA, a Military Type Certificate (MTC) must be issued. Similar to an 
FAA Type Certificate, a MTC is a document that provides evidence that an aircraft 
system type design is in compliance with its approved certification basis. An Air Force 
official identified as a Technical Airworthiness Authority (TAA) is authorized to approve 
the certification basis (set of approved criteria, standards, and methods of compliance) 
and issue MTCs based on defined airworthiness standards. Following issuance of a MTC 
and evidence of safe operation, program managers issue a Military Certificate of 
Airworthiness (MCA) to each individual aircraft (AFPD 62-6, 2010). AFI 62-601, USAF 
Airworthiness (June 2010), which outlines the military certification process, states that 
the MCA “remains in effect for the approved service life as long as the air system 
configuration is in a condition for safe operation (i.e., properly maintained in accordance 
with approved maintenance documentation, and the system is operated in accordance 
with the approved flight manual and within the approved mission usage).” AFI 62-601 
applies to all aircraft organizationally operated by components of the USAF. 

Detailed criteria used in the detennination of airworthiness for all manned, 
unmanned, fixed and rotary wing air vehicle systems are contained in MIL-HDBK-516B. 
This handbook not only identifies the DOD and military document references for each 
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requirement, but in many cases identities the FAA counterpart to those documents as 
well. These dual references provide an indication that military certification requirements 
are often based on existing FAA requirements, and the two processes closely mirror one 
another. 

Potential Costs of FAA Certification 

Initial Certification 

Typically, since MCDA are based off of previously certified aircraft designs and 
are already in production, they have already been certified by the FAA (Humerick & 
Minnich, 1992). As such, the question for decision makers is not whether to obtain FAA 
certification, but whether to maintain FAA certification (and for how long). This initial 
FAA certification is required in order for the aircraft to be operated and tested prior to 
being turned over to the military organization that purchased the aircraft. In some regard, 
because the costs associated with initial certification of MCDA have been paid, they 
might be considered sunk costs, or costs that are lost forever once they are paid (Baye, 
2010). In other words, the military organization that purchased the MCDA would have to 
pay these costs regardless of whether the decision maker opted to keep the certification or 
not (Stockman et ah, 2011). 

In addition to the sunk costs associated with obtaining FAA certification are the 
fixed costs associated with maintaining it. Fixed costs do not vary with output and can 
include one-time purchases or upgrades (Baye, 2010). Costs associated with maintaining 
FAA certification that fall into this category might include those associated with updating 
military processes and procedures to allow for commercial parts pooling. For example, 
the costs associated with changing supply manuals and supply databases to enable 
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traceability of FAA approved aircraft parts might be substantial at first, but then might 
then reduce as military processes began to mirror those of the FAA. 

Sustainment Strategy Dependent Costs 

The costs associated with maintaining FAA certification are also heavily 

dependent on the overall sustainment strategy that will be used for the MCDA. Some 

sustainment strategies include organic (sustainment supported by U.S. government 

entities, such as DOD organizations), contractor logistics support (or CLS, or contractor 

provided sustainment), or a mix of the two. Another sustainment strategy is to transition 

from one type of support to another over the life cycle of the aircraft, such as starting 

with contractor support and transitioning to organic. These strategy choices are often 

influenced by other, much broader DOD requirements, such as the “50/50” requirements 

outlined in 10 U.S.C. Section 2466. This section of the code states the following 

regarding depot support for military aircraft: 

Percentage Limitation: Not more than 50 percent of the funds made available in a 
fiscal year to a military department or a Defense Agency for depot-level 
maintenance and repair workload may be used to contract for the performance by 
non-Federal Government personnel of such workload for the military department 
or the Defense Agency. Any such funds that are not used for such a contract shall 
be used for the performance of depot-level maintenance and repair workload by 
employees of the Department of Defense. (10 USC Sec. 2466, 2011:1) 

In short, no more than 50% of depot funding may be used for CLS. Another significant 

influence on choice of sustainment strategy involves the identification of specific 

capabilities as core requirements, as defined below. 

Core: The depot maintenance capability (including personnel, equipment, and 
facilities) maintained by the Department of Defense at Government-owned, 
Government-operated facilities as the ready and controlled source of technical 
competence and resources necessary to ensure effective and timely response to a 
mobilization, national defense contingency situations, and other emergency 
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requirements. Depot maintenance for the designated weapon systems and other 
military equipment is the primary workload assigned to DOD depots to support 
core depot maintenance capabilities. (DODI 4151.20, 2007:5) 

The impact of the core identification label is simple: if the capability is identified as a 

core requirement, it must be maintained by government owned and operated facilities. 

Thus, regardless of which sustainment method may be the most cost effective, core 

requirements and 50/50 limitations can push sustainment decisions toward implementing 

less than optimal solutions. 

Other Costs 

Other costs associated with FAA certification (and the purchase of commercial 
derivatives in general) are numerous and are detailed in published guides for commercial 
derivative acquisition and commercial-off-the-shelf (COTS) purchasing. For example, as 
outlined in Gansler and Lucyshyn’s “Commercial-Off-the-Shelf (COTS): Doing it 
Right” (2008), there are many challenges to consider before making COTS purchases. 
Item availability may be jeopardized as commercial vendors go out of business. 
Unanticipated technology changes can also lead to increases costs. Vendors may also 
substitute parts of a previously qualified component, creating a requirement for 
requalification. Additionally, COTS purchases may “lock the user into a proprietary 
technology... if an organization adapts a commercial ERP system, transitioning to another 
developer’s ERP may be very costly” (Gansler & Lucyshyn, 2008:56). 

The nature of the various costs mentioned in this chapter as being sunk or fixed is 
one of the reasons this research focuses on optimizing benefits as opposed to perfonning 
a specific cost-benefit analysis. Additionally, many of the costs are extremely situation- 
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specific and likely to change considerably on a case-by-case basis. As such, they are not 
conducive to being included the generic MADM support tool presented in Chapter Three. 

Potential Benefits of FAA Certification 

Participation in the Commercial Parts Pool 

Perhaps one of the most significant benefits that comes from maintaining similar 
certification of MCDA to commercial fleets is the availability of the commercial parts 
pool. A parts pool is essentially a supply of parts that has been established for the use by 
a group or fleet of aircraft. By establishing a common pool of parts that all users can tap 
into, the level of spare parts inventory that each individual member must maintain can be 
reduced. This includes inventories for both operational use as well as safety stock. In 
order to participate in the commercial parts pool however, very specific FAA guidelines 
regarding repair and tracking of parts must be followed (Bongiovi et al., 2009). 
Specifically for MCDA, FAA certification must be maintained and participants must 
maintain the same airworthiness standards and criteria for those parts (Stockman et al., 
2011). Specific guidance from the FAA on the subject of approving aircraft parts and 
parts-pooling states the following: 

An operator must ensure that all replacement parts meet or exceed original 
certification standards. Standard hardware and materials can be used and 
exchanged without special procedures. When special requirements must be met, 
accurate documentation must be maintained. Purchase, use, and exchange of 
parts require special procedures that must be part of the operator's manual, and in 
certain circumstances, part of the operator's operations specifications. (FAA 
Order 8900.1, 2007:2) 

Further guidance can also be found in FAA Advisory Circular AC 20-169 which 
states that new, modified, or replacement parts produced for sale must be approved under 
the Type Certificate, a production certificate, a technical standard order authorization 
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(TSOA) or FAA letter of TSO design approval. It is absolutely critical that configuration 
control and proper tracking be maintained on all parts in the pool. For example, in order 
for used MCDA parts to be accepted back into the commercial parts pool, it must be 
shown that configuration control has been maintained and that all applicable 
requirements outlined in Title 14 Code of Federal Regulations Part 21 “Certification 
Procedures for Products and Parts” have been complied with. 

There are several parts-pooling strategies that have been implemented in the past. 
One strategy is based on a one-way flow of parts, where FAA compliant parts can be 
purchased from a commercial pool and installed on USAF aircraft. These parts are not 
returned to the commercial parts pool once used because the aircraft or repair process 
involved did not meet FAA requirements. In contrast to this strategy is the two-way parts 
flow model which allows for shared use of FAA compliant parts (also known as “parts- 
swapping”). Other strategies include the Foreign Military Sales (FMS) Virtual Fleet 
model which allows for full parts-sharing among C-17 aircraft customers, for example, 
for parts managed by Boeing. The FMS Cooperative Logistics Supply Support 
Arrangement (CLSSA) model is an agreement between USAF and a foreign military 
service that establishes terms and conditions for follow-on spares support (AFMAN 23- 
110, 2009). Essentially, maintenance of the FAA type design (retention of the Type 
Certificate) can enable participation in these types of spare parts pools, which in turn can 
lead to decreased inventory costs and reduced aircraft downtime (Bongiovi et al., 2009). 

Use of Existing Commercial Resources and Support 

The benefit being described here is an extremely broad category that includes 
access to and utilization of commercial maintenance and supply resources, as well as 
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aircraft inspection and servicing capabilities. While utilization to existing supply and 
maintenance infrastructures is a benefit that can be associated with MCDA programs 
regardless of FAA certification, it should be included in the decision analysis as it can 
impact how long FAA certification should be maintained. Other important aspects of this 
benefit category are existing engineering resources that may be available in the 
commercial market for MCDA. Previous reports have identified that our government’s 
continued reliance on its own engineering resources is less efficient than utilizing existing 
FAA procedures and personnel already available via commercial engineering support 
offices (Ross, 2010). 

Access to Existing Commercial Data 

This type of commercial data being referred to here is primarily the technical data 
used for modification (hardware, software, test data, design data, etc.). Access to this 
data requires negotiation of data rights with the original equipment manufacturer (OEM). 
This can be a fairly contentious issue during MCDA contract negotiations because the 
DOD typically wants to be able to modify their aircraft if they choose to do so, and 
believes that the data rights should be included in the purchase price. The subject of data 
rights is typically one of the most challenging with regard to MCDA programs (Bongiovi 
et ah, 2009). If access to this type of data is not negotiated properly, costly reverse¬ 
engineering may have to be accomplished in order to recreate the manufacturing 
specifications and technical drawings. 

Use of Existing Processes 

Some of the existing processes in use by the FAA have proven to be very 
successful over time. Examples of the types of processes being referred to in this section 


22 



include FAA Service Bulletin and Airworthiness Directive processes, repair and 
inspection processes, as well as many other maintenance or safety related processes that 
have been proven in the commercial industry that may be applied to military operations. 
Sale of Demilitarized MCDA to Commercial Market 

The secondary market for previously owned military aircraft is dependent upon 

many factors such as aircraft type, age, fleet size, availability of similar aircraft, etc. If 

the sale or transfer of MCDA back to the commercial market is to be considered part of a 

budgeting strategy, the commercial value of the aircraft must be fully understood. 

Additionally, these aircraft must be maintained in a manner that preserves nearly 

identical functionality with to their commercial counterparts (“Buying Commercial,” 

2009). Maintaining FAA certification of MCDA is a big part of ensuring commercial 

commonality, as outlined in the following FAA Order excerpt. 

Ex-military aircraft now under civilian type certificates create parts problems, 
particularly when the original manufacturer has ceased production. Certain parts 
of original manufacture are available for a given aircraft for a number of years 
after its departure from military status. If original manufacturer fabrication can be 
substantiated for such parts, they are acceptable providing they comply with all 
applicable airworthiness directives. Certain parts for ex-military or currently 
manufactured aircraft are and have been scarce. Occasionally, parties other than 
the original or approved manufacturer produce these (FAA Order 8900.1, 2007) 

Some of the processes and potential problems associated with demilitarizing 

military equipment are detailed in a Congressional Research Service (CRS) report to 

Congress from 2006. In this report, “Demilitarization of Significant Military 

Equipment,” some of the issues identified included that the DOD had incorrectly coded 

military components, resulting in the sale of items that should never have been made 

available to the public. Obviously, extreme diligence must be exercised in carrying out 
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this disposal alternative, but if done successfully, it can be less costly than other 
alternatives. Ultimately, if it is feasible to dispose of excess DOD inventory in this 
manner, maintaining FAA certification may make this option less cost prohibitive. 
Configuration Management 

FAA regulatory and advisory guidance regarding aircraft configuration 

management, control, and tracking are extremely comprehensive. Details regarding 

evaluation and verification of aircraft configuration are outlined in FAA Order 8900.1, as 

well as several sections of the CFR. Some of the core issues with regard to aircraft 

configuration include verifying that the aircraft conforms to FAA approved type design, 

that it is a safe operating condition, and that substantiating records exist to show 

compliance with the operational compliance with Title 14 of the CFR (FAA Order 

8900.1, 2007). While USAF configuration management is also well established, it can be 

more difficult to maintain due to the sheer number of different aircraft configurations in 

use. Some of the reasoning behind these difficulties was discussed in Stockman, Ross, 

Bongiovi, and Sparks’ “Successful Integration of Commercial Systems: A Study of 

Commercial Derivative Systems” as outlined below. 

In some instances the Government has purchased certain CDA assets in very 
limited quantities, often including as few as one aircraft of a type per year. This 
practice, though sometimes necessary due to congressional appropriation 
constraints, results in higher costs for each individual CDA purchase and the 
creation of what the CDA industry terms —partial orphan aircraft. That is, the 
configuration of most CDA systems in continuous production evolves on an 
annual or even more frequent basis, and this causes the Government 1 s —one-at-a- 
time purchasing practice of CDA systems over many years to put aircraft in the 
military serviced fleets that are slightly or even significantly different in 
configuration (Stockman et ah, 2011:179). 
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It is possible that increased FAA oversight in this area would potentially lead to process 
improvements and decreased risk from improved configuration control and 
documentation. 

Quality and Safety 

While the benefit of additional FAA oversight in the areas of quality and safety 
can be difficult to quantify, the sharing of best practices between similar organizations 
can lead to substantial cost savings and improved efficiencies. One of the major roles of 
the FAA is to promote safety through regulation of civil aviation. Additionally, FAA 
quality assurance is often used as the benchmark for military quality improvement 
efforts. The many years of experience the FAA has as an organization in these areas can 
bring substantial benefits. However, difficulty in quantifying these benefits often leads to 
underestimating the positive impact of maintaining a close cooperative partnership with 
the FAA. 

Other Advantages to FAA Certification 

As previously identified in this chapter, the potential benefits associated with 
maintaining FAA certification of MCDA are many, and this research effort does not 
represent an all-inclusive list, but there are a few more that deserve mention. For 
example, MCDA purchased by the USAF typically come with FAA certification as part 
of the purchase price. If major modifications are not made on the production line, the 
FAA Type Certification enables the military certification process to be completed more 
quickly, relative to aircraft that have not already been certified by the FAA. This allows 
for shorter turn-around time from delivery to operation. 
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Looking at the aviation industry from a broader perspective, some of the benefits 
to FAA certification of MCDA can also be viewed as they would apply to the 
international aviation industry. In other words, maintaining FAA certification for civil 
and military aircraft may represent a step toward establishing international airworthiness 
criteria that would be accepted industry wide. A study commissioned by the President of 
the United States and Congress in 2001 on the future of the aerospace industry 
highlighted several points on this subject in their final report, published in 2002. Some of 
these points are identified below. 

1. Technology development and associated product cycle times have outpaced 
the applicable FAA regulations, policy, guidance and oversight capacity. 

2. The time and cost to market for new technology communication, navigation, 
surveillance and air traffic management (CNS/ATM) products is prohibitive 
to the FAA’s National Airspace System modernization plans and priorities. 

3. The lack of international agreements concerning the interoperability of 
CNS/ATM products and the harmonization of applicable regulations is a 
barrier to defining International Airspace System U.S. Air Transportation 
System (INAS) operations and to any significant development or certification 
cost efficiencies for the associated products and systems. 

4. Current methods, policies and practices do not support the types of operations 
necessary for efficient use of the INAS by the aviation community. 
(“Commission on the Future,” 2002:pp. 2-4) 

Based on these findings, the study went on to conclude the following: 

To transform our air transportation system, government and industry must work in 
partnership to enable certification regulations and processes that keep pace with 
advancing technical innovations. We must be able to efficiently certify the airborne 
information technologies, integrated systems, and communications li nk s that will 
comprise our future system. (“Commission on the Future,” 2002:2-5) 

Methodology to Obtain Decision Support Tool Criteria Consensus 

Various methods were considered regarding obtaining the right criteria to include 
in the decision tool. Initially, basic research into existing studies and reports was 
considered an appropriate method of detennining the primary costs and benefits 
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associated with maintaining FAA certification of MCDA. Another method considered 
was to utilize a single advisor or expert in the subject matter to attempt to validate 
research findings. While this second method may have resulted in a certain level of 
uniformity in the responses, it was difficult to identify a single individual that had 
experience over all areas of the extremely broad topic of military aircraft acquisition and 
certification. Ultimately it was decided that, due to the extremely subjective nature of the 
research and decision criteria being considered, the most effective and efficient approach 
would be to achieve a consensus of subject matter experts through implementation of a 
Delphi study. 

Delphi Study Techniques 

By definition, the Delphi technique is “a series of repeated interrogations, usually 
by questionnaire, asked of individuals whose opinion or judgments are of value. The 
purpose is to arrive at a consensus regarding an issue under investigation.” (Cornell, 

1980:171) In short, a Delphi study is a technique used to elicit a group of responses from 
a panel of experts. The technique is very useful for instances in which quantification and 
analysis are difficult, and the approach to validating data will be primarily a judgmental 
one. In past applications, it has proven to be a useful method of pooling the knowledge 
and experience of participants in an anonymous way (Cornell, 1980). In fact, it is 
perhaps the anonymous aspect of the Delphi study that enables quality inputs to be 
received. Anonymity helps reduce the effects of socially dominant individuals or 
pressure from supervisors and peers to confonn to a particular viewpoint (Dalkey, 1967). 
Essentially, the Delphi method replaces direct, face-to-face debate with “individual 
interrogations (best conducted by questionnaires) interspersed with information and 
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opinion feedback derived by computed consensus from the earlier parts of the program” 
(Brown, 1968:2). 

Along with anonymity, there are two additional characteristics of a Delphi study 
that make it distinctive from other methods. One of these characteristics is controlled 
feedback. The tenn “controlled feedback” encompasses the method within a Delphi 
study wherein data collected over several iterations is summarized and “fed back” to 
participants in an effort to achieve consensus or at least reduce the level of disparity 
between responses. The third characteristic of a Delphi study is the idea of a statistical 
index or representative group opinion. This group opinion is typically the result of 
several iterations conducted with participants (Dalkey, 1967). 

Typically, the Delphi technique is considered to have either two or four phases. 
Both approaches are similar, with the difference being that with the four-phase approach, 
the two primary phases (exploration and evaluation) are simply broken down further into 
four distinct steps. For example, the first three phases of the four-phase approach can be 
considered exploratory in that each participant contributes information (Phase 1) which is 
then clarified through iterations or feedback (Phase 2). If there are significant differences 
or disagreements then those areas are further explored to determine the underlying 
reasons behind them (Phase 3). Finally, evaluation occurs when all inputs are analyzed 
and fed back to the participants for consideration (Phase 4) (Linstone & Turoff, 1975). 

Participant selection and panel size are important considerations to the Delphi 
technique. Delphi participants typically fall into one of three categories: stakeholders, 
experts, and facilitators, as defined here by Linstone and Turoff s “The Delphi Method: 
Techniques and Applications” (1975). 
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Stakeholders: Those who are or will be directly affected by outcomes or 
decisions. 

Experts: Those who have an applicable specialty or relevant experience. 

Facilitators: Those who have skills in clarifying, organizing, synthesizing, or 
stimulating. 

Most panels consist of a mix of the three categories, and the proportion of each will vary 
from application to application, but a common trait throughout the group will likely be 
some level expertise in the subject of the study. 

With regard to panel size, often one of the most critical criteria is the study 
timeline. As the size of the panel increases, the time allowed to each member to evaluate 
and respond decreases. This is primarily due to the fact that the level of complexity in 
summarizing and providing feedback for multiple iterations increases with panel size 
(Linstone & Turoff, 1975). 

The key advantages to the Delphi study, anonymity and convenience, do not come 
without some trade-offs. Response levels are often affected by participant attrition, 
participant workload, and overall willingness (or lack thereof) to submit responses. 
Additionally, it is important to understand that the Delphi approach does not always 
result in consensus of the participants. While responses can be vague and subjective 
compared to other methods, it has been shown that often, the scientific technique behind 
many theories or hypothesis is often a somewhat philosophical basis (Linstone & Turoff, 
1975). More important than reaching consensus is capitalizing on the level of expertise 
of the qualified participants and achieving a tool or model that is of value to the decision 
maker. 
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Multiple Attribute Decision Making (MADM) 

Problems that involve multiple, conflicting criteria can become very complex as 
the number of criteria increases. In these types of situations, it can be helpful to have a 
model or tool that categorizes or prioritizes the criteria in some way. Multiple Attribute 
Decision Making (MADM) addresses this issue by helping to establish a method of 
“making preference decisions (e.g., evaluation, prioritization, selection) over the 
available alternatives that are characterized by multiple, usually conflicting attributes” 
Hwang and Yoon, 1995:2). As mentioned previously, the decision tool produced from 
this study utilizes some of the principles of MADM. The decisions or problems 
addressed by MADM can be very diverse, but a summary of some of the common 
characteristics, taken from Yoon and Hwang’s “Multiple Attribute Decision Making: An 
Introduction” (1995) are outlined in Table 3. 

In order to apply MADM, decision makers must often assign weights to the 
various attributes involved in their decision. These weights may be assigned in 
accordance with an ordinal scale (weighted by rank, as in 1 st , 2 nd , 3 rd , etc.) or cardinal 
scale (weighted by a mathematical value), and while ordinal scales are usually simpler to 
assign, cardinal scales are preferred for MADM. Cardinal weights are typically 
normalized to sum to one. Through these methods of weighting, fairly subjective or 
qualitative attributes can be roughly quantified. For example, when applying weights to 
perceived benefits from a certain decision, greater weight value implies greater 
preference (Hwang &Yoon, 1995). While a simple application of relative weight based 
on preference or perceived value may seem arbitrary, such value judgments may be 
necessary, especially for complex problems. In fact, some problems even require such 
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value judgments in order to gain important insights that can only be achieved through 
value-focused thinking, or deciding which attribute is best and making the necessary 
choices to capitalize on that attribute (Keeney, 1992). 


Table 3: Common M 

ADM Characteristics 

Characteristic 

Description 

Alternatives 

A finite number of alternatives, from 
several to thousands are screened, 
prioritized, selected, and/or ranked. The 
tenn “attribute” is synonymous with 
“option,” “policy,” “action,” or 
“candidate,” among others 

Multiple Attributes 

Each problem has multiple attributes. A 
decision maker must generate relevant 
attributes for each problem setting. The 
number of attributes depends on the nature 
of the problem. The term “attributes” may 
be referred to as “goals” or “criteria.” 

Incommensurable Units 

Each attribute has different units of 
measurement. For example, time or money 
may be expressed as units while safety or 
quality improvements might be expressed 
in a nonnumeric way. 

Attribute Weight 

Information regarding the relative 
importance of each attribute. Weights can 
be assigned or developed by methods 
utilizing ranks or ratios. 

Decision Matrix 

A MADM can be concisely expressed in a 
matrix fonnat, where columns indicate 
attributes considered in a given problem 
and rows list competing alternatives. 


For some decisions, applying cardinal weights in this manner allows for a more 
detailed look at the attributes. Not only are the attributes considered in order of 
precedence, but they are also considered by level of importance or impact as they relate 
to one another. Furthermore, by looking at a selection of attributes considered to have 
the most impact, they can be viewed in a simultaneous and hierarchical way. In other 
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words, sometimes it is not enough to simply compare two attributes as a choice of two 
alternatives. Sometimes in MADM applications, it may be necessary to consider all 
attributes at the same time in order to be able to make the necessary trade-offs between 
the available choices (Holloway, 1979). 

Summary 

In order to analyze complex decisions such as whether or not to maintain FAA 
certification of MCDA, decision makers must fully understand the background of the 
question as well as the process behind MCDA acquisition and certification. With so 
many variables to consider, it can be difficult for any single individual to fully grasp all 
aspects of the problem to the extent necessary to come to an optimal solution. For this 
reason, these types of decisions are seldom left to a single decision maker and are instead 
decided upon following detailed input from many experts from many different fields. 
This approach to decision making and weighing costs and benefits is based on some of 
the principles of the Delphi technique and MADM support tool utilized in this study. 
These are the methods outlined in Chapter Three that will help answer the following 
research questions: 

Question 1: What are the decision criteria USAF leaders should consider when 
determining whether or not to maintain FAA certification of MCDA? 

Question 2: How do the costs and benefits associated with maintaining FAA 
certification of MCDA change over time? 

Question 3: If the choice is made to maintain FAA certification of MCDA, how 
long should it be maintained? 
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The answers to these questions, as based on the research and inputs received from subject 
matter experts, are provided in Chapter Four. These results are the basis for the 
conclusions and recommendations made in Chapter Five. 
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III. Methodology 


Overview 

This research attempts to answer the question of whether or not to maintain FAA 
certification of MCDA, and if so, for how long? In order to answer this question, several 
objectives must be met. First, the primary criteria that a decision maker uses to evaluate 
possible options must be established and well-defined. Next, those criteria must be 
ranked or assigned a level of importance relative to one another in order to detennine 
their value when considering trade-offs. The final objective, detennining attribute 
behavior over time, is perhaps the most difficult to identify due to its extremely 
subjective nature. It is fairly straightforward to identify the most important criteria in the 
FAA certification decision making process and then to rank or assign weight to each, 
however, once those criteria are identified and prioritized, they do not simply remain 
constant with respect to time. Therefore, it must also be an objective of this research is to 
show how the decision criteria behave or change over the lifecycle of the MCDA fleet. 
Delphi Study 

Objectives 

There were several purposes for the Delphi study used in this research effort. The 
first was to elicit inputs from subject matter experts (SMEs) regarding what 
considerations a decision maker should make when determining if a commercial 
derivative was the right choice for a particular military acquisition. Additionally, it 
would be important to understand where a decision maker might perceive the greatest 
amount of risk regarding that initial commercial derivative purchasing decision. 
Following the decision to utilize a commercial derivative, SME inputs would also be 
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needed to determine what considerations a decision maker should make regarding 
maintaining FAA certification, and if there were any li nk s between the risks associated 
with purchasing commercial derivative and those associated with maintaining FAA 
certification. This was an important aspect of the first Delphi questionnaire because 
initial research identified that some of the benefits associated with maintaining FAA 
certification may be perceived as an offset to the risks associated with using MCDA in 
general. 

Another important objective of the Delphi study was to establish weight or impact 
level of the criteria involved in the FAA certification decision. While there are many 
methods of assigning such weight, the method chosen for this study was a simultaneous, 
non-hierarchical method in which a set amount of points (in this case, 100 points) were to 
be distributed among each of the criteria. The method was simultaneous in that all 
criteria were considered at once (as opposed to pairwise considerations of two criteria). 
Simultaneous weighting was used to eliminate the possibility of the participant providing 
conflicting weights, as can sometimes happen in a pairwise comparison. For example, in 
a pairwise comparison of several items it would be possible to assign weights such that 
Attribute A was weighted higher than Attribute B, and Attribute B higher than Attribute 
C, but then Attribute C contradictorily weighted higher than Attribute A at a later point in 
the weighting process. With simultaneous weighting of all attributes at once, this risk of 
this type of error is greatly reduced. The attribute weighting method was considered non- 
hierarchical in that there were no sub-categories of decision criteria, only a list of primary 
considerations or benefits. The attributes were listed in this manner because they were 
identified as the primary criteria in the decision process and as such, were meant to be 
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considered separately. Additionally, weights assigned in this type of non-hierarchical 
fashion can then be normalized to show their relationship with one another, as was done 
in this study. 

Some level of bias was likely introduced into these weights based on participant 
background and behavior, and there are more complex methods of attribute weighting 
that may have been able to reduce some of this bias. However, for some multiple 
attribute decisions, the type of weighting method used is not nearly as important as 
participant selection and method of elicitation. There are many weighting methods that 
can be equally successful, so long as the attributes and the reason for applying weights 
are clearly understood by the study participants (Poyhonen, 1998). For this study, it was 
detennined that using a more complex weighting method would have been only 
marginally more beneficial when compared to the amount of extra time needed to elicit 
responses. Taking into consideration the relatively short amount of time allotted to 
complete this study, as well as the high operations tempo and workload of the 
participants, the researcher instead decided to place extra focus on correctly identifying 
the attributes themselves, and ensuring the questionnaires were simple, easy to 
understand, and could be completed fairly quickly. 

In order to conduct this type of study, the researcher completed the Human 
Subjects Research Training basic course through the USAF Research Laboratory. 
Additionally, because no demographic or sensitive data was collected during the course 
of the study, an exemption to the Federal, DOD, and USAF human experimentation 
requirements was approved (see Appendix P for Exemption Approval Letter). 
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Delphi Study Round One 

There were three general purposes for the first round of study questionnaires 
(Appendix B): 

1. Identify decision criteria used in determining whether or not a commercial 
derivative is the right acquisition choice 

2. Identify the decision criteria used in detennining whether or not to maintain 
FAA certification of MCDA 

3. Identify the primary risk concerns a decision maker faces when making the 
decisions identified in items 1 and 2 

Delphi Study Round Two 

There were two general purposes for the second round of study questionnaires 
(Appendix E): 

1. Assign simultaneous, non-hierarchical weights to the primary FAA 
certification decision criteria identified in Delphi study round one 

2. Identify how FAA certification decision criteria behave or change over the 
lifecycle of the MCDA (based on the behavior curves provided in the 
questionnaire) 

Participant Selection 

In detennining the size and make-up of the panel of participants, several criteria 
were considered. With regard to the panel’s general make-up, two types of individuals 
were considered to be important to obtaining the type of feedback necessary for the topic 
of maintaining FAA certification of MCDA - experts and stakeholders. Experts are 
those individuals considered to have knowledge or experience relative to the subject, 
while stakeholders may be considered as those individuals who will be affected by the 
decisions made (Linstone & Turoff, 1975). For this panel, experts were chosen based on 
their knowledge and experience in USAF aircraft acquisitions and certification. 
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Stakeholders were chosen based on their involvement in current operations involving 
FAA certified aircraft or repair processes. In detennining the optimal size of the panel, 
consideration was given to the overall timeline for the study while also keeping in mind 
that there is a point at which increasing the number of participants provides only marginal 
benefit to the relative value of the information provided. Additionally, examples of this 
style of Delphi study (for somewhat smaller organizations) consisted of panels of four to 
eleven participants. 

Based on these factors, it was determined that approximately 20 participants 
would be sufficient to gather the infonnation needed and an initial request for volunteers 
was submitted to four organizations. Initial responses received as a result of the research 
request memo (Appendix O) identified 18 volunteers, most having backgrounds in USAF 
acquisitions or experience with MCDA maintenance and operational processes. Positions 
held ranged from organizational to MAJCOM levels (Appendix Q), and most individuals 
had at least five years of experience in their current positions (with some having as many 
as 15 years). However, due to participant attrition over the range of the study, the final 
participant pool consisted of 13 individuals for round one (questionnaire one) and 10 
individuals for round two (questionnaire two). Reasons for this attrition included 
participant retirement as well as failure to return completed questionnaires. Multiple 
attempts were made to re-engage with participants that did not return questionnaires (or 
returned incomplete questionnaires), but these attempts were largely unsuccessful. It is 
the opinion of the researcher that a high operations tempo and heavy workload may have 
been contributing factors to participant attrition. Those participants that provided 
incomplete responses (or no responses) were not included in the final results. 


38 



Delphi Study Design 

Both Delphi questionnaires were administered via email to individual participants. 
Individual emails were used (as opposed to group or mass emails) in order to reduce the 
likelihood that the participants would discover the identities of the rest of the group. This 
would then potentially also reduce the likelihood that participants would communicate 
with (and possibly influence) one another. Each questionnaire was administered two 
times, with the second iteration incorporating some of the inputs from the first iteration in 
an effort to increase the level of concurrence. Due to the fairly short timeframe for the 
study, follow-up contact by phone was also made to ensure questionnaires were received 
and then returned in a timely manner. 

The first questionnaire used for round one of the Delphi study included an 
introduction to the Delphi method, as well as a brief summary of the overall purpose of 
the research and statements outlining voluntary participation and confidentiality. As 
mentioned, the purpose of the first questionnaire was primarily to identify criteria and 
risks associated with USAF acquisition of MCDA and subsequent FAA certification. 

The questionnaire contained three statements that were followed by five questions 
that attempted to identify the participant’s stance or beliefs about those statements. The 
first two statements were a comparison of potential risks associated with the purchase of 
commercial derivatives early in the production cycle versus a purchase made later in 
production. The items of risk identified in these first two statements were a culmination 
of research into current sources on the subject. Questions #1 and #2 simply asked if the 
first two statements accurately represented the primary risks associated with USAF 
purchase of commercial derivatives, and if not, what items the participant felt should be 
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added or removed. Throughout the initial research for this study, a common theme had 
emerged that the benefits associated with maintaining FAA certification of MCDA was 
only relevant if there was a fairly large commercial fleet in use, and Question #3 simply 
asked if the participants agreed with this theme, and if not, why they disagreed. The final 
two questions (#4 and #5) on questionnaire one were developed as an effort to allow 
participants to provide open input regarding USAF purchase of commercial derivatives 
and subsequent FAA certification. Question #4 asked participants to provide comments 
or recommendations they would suggest to a decision maker regarding the decision to 
utilize commercial derivatives for military use. Question #5 similarly solicited 
suggestions participants would provide decision makers regarding the decision to 
maintain FAA certification. Overall, responses to these five questions were used to 
establish a list of criteria or attributes regarding benefits associated with FAA 
certification of MCDA. 

The focus of the second questionnaire was to rank the attributes identified in 
questionnaire one and attempt to determine how those attributes might change over time. 
First, the attributes that were identified from questionnaire one were listed in no 
particular order. Participants were then asked to distribute 100 total points across all of 
the attributes, assigning points based on level of perceived impact. Participants were 
instructed to use all 100 points (no more and no less) in their effort to apply weights to 
the attributes. This method was chosen in order to not only allow the attributes to be 
ranked (1 st , 2 nd , 3 rd , etc.), but also to enable participants to distinguish various levels of 
importance. In other words, if participants felt that a certain attribute was extremely 
important or had a high level of impact, they could assign an appropriately high number 


40 



of points. For example, in a simple ranking, the relative difference between the first and 
second attributes is the same as the difference between the fifth and sixth. However, by 
utilizing an allowance of points, the difference between the first and second attributes 
might be 30 points, while the difference between the fifth and sixth might only be 5 
points. This method of attribute weighting is a simple but effective means of identifying 
to a decision maker which attributes have the biggest impact and on what scale, relative 
to the rest of the attributes. 

The focus of the second part of questionnaire two was to ask participants to 
identify how the attributes from the first part of the questionnaire behaved over time. In 
this questionnaire, participants were asked to consider a timeline that represented the 
lifecycle of an arbitrary fleet of aircraft that were already in production. A graphic 
timeline was provided that divided aircraft lifecycle into six segments. The segments 
were identified using common terms that all participants would be familiar with based on 
their backgrounds, and were labeled as follows: 

1. Early Production 

2. Mid Production 

3. Late Production 

4. Post Production 

5. Commercial Retirement 

6. MCDA Retirement 

While an aircraft lifecycle typically begins much sooner than “early production” 
(such as the identification of a need for a particular capability, for example) and then 
progresses through various stages of design and requirement identification before 
entering production, MCDA can be viewed slightly differently. MCDA are by definition 
derivatives of previously developed commercial aircraft that are used for military 
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purposes, and as such, the lifecycle for use in this questionnaire begins with “early 
production” as opposed to “identification of need.” 

In order to simplify the identification of attribute behavior with respect to time (or 
lifecycle), participants were provided with nine figures that each depicted an impact 
curve plotted against a timeline. The Y-axis or vertical axis of each graph showed level 
of impact, and the X-axis or horizontal axis of each graph represented time. Axis scale 
and label were designed to be extremely simple in order to make the graphs as generic as 
possible. For example, the level of impact (or Y-axis) had no numerical scale, and was 
simply labeled to show that the lower portion of the axis represented less impact while 
the upper portion of the scale represented more impact. Similarly, the timeline (or X- 
axis) had no numerical scale, but simply two points (A and B) that corresponded to 
specific points in the applicable curve, meant to be chosen from the provided timeline. A 
brief description of how the curves represented behavior over time was provided 
underneath each graph. The entire selection of nine graphs can be found in Appendix E, 
but a few examples are provided on the following pages for clarification. 



Figure 1. Example of Exponentially Decreasing Impact over Time 
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Figure 1 is an example of the graph provided to participants to represent a 
decreasing attribute impact over time. In this case, an exponential curve is used to show 
a fairly constant rate between points A and B, followed by a dramatic decrease after point 
B. Participants were asked to select two points from the six identified in the lifecycle 
timeline that best represented points A and B on the graph. For example, if one of the 
MCDA FAA certification decision criteria or attributes was the benefit associated with 
being able to participate in a common spare parts pool with the commercial industry, 
participants might identify this graph as a close representation of how the impact of those 
benefits would change over time. One might say that the impact of that benefit would be 
felt rather quickly once the commercial fleet reached mid-production, and then fall off 
dramatically once the commercial fleet retired. In that case, point A on the X-axis would 
represent “mid-production” (or “2” on the lifecycle timeline) and point B would represent 
“commercial fleet retirement” (or “5” on the lifecycle timeline). Once the participant 
selected the appropriate behavior graph and identified where points A and B fell on the 
lifecycle timeline, they entered that information into the table provided within the 
questionnaire. If the attribute impact increased in a linear manner or behaved in an 
exponentially increasing manner, then the participant might choose one of the graphs 
shown in Figures 2 and 3. 
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More 


Less 



Fairly constant rate of increase 
through all points 


Figure 2. Example of Linearly Increasing Impact over Time 



Figure 3. Example of Exponentially Increasing Impact over Time 


Linear and exponential curves were chosen because they represented fairly 
common attribute behaviors. Additionally, there were curves provided in questionnaire 
two that represented a normal curve (increased to a peak and then decreased or visa- 
versa) as well as a flat line that represented little or no change over time. In an effort to 
maintain simplicity of the questionnaire, these several curves were determined to provide 
an adequate representation of attribute behavior for the purposes of this study, even 
though any number of additional curve types could have been provided. As can be seen 
in Figure 2 above, the linear representation does not have points A and B listed on the 
graph because it is meant to represent a constant rate of increase through all lifecycle 
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timeline points. Similarly, the linearly decreasing graph and flat (no change) graphs did 
not require participants to identify points A and B from the timeline. 

Lastly, questionnaire two provided one final opportunity for participants to 
modify the list of attributes that had been previously identified in questionnaire one by 
asking them to view the list one last time and update areas as they felt necessary. This 
was meant to allow for finalization of the attribute list, in that participants could add, 
change, or remove items of their choosing if they believed items were incorrectly worded 
or falsely identified. Questionnaires one and two are located in Appendices B and E, 
respectively. 

MADM Based Support Tool 

Objectives 

In order to graphically display the infonnation collected by the Delphi study, a 
MADM based support tool was created (Appendix J). The tool is a simple model that 
couples attribute weights with behavior over time to produce an “impact score” for each 
of the six lifecycle points identified earlier. These scores were then converted to an easy 
to read bar chart identifying the relative benefit comparisons across the proposed fleet 
lifecycle (Appendix K). In short, the tool provides a decision maker with an approximate 
estimate of where he or she is likely to achieve the most benefit from maintaining FAA 
certification and at what point those benefits taper off. It also allows the decision maker 
a glimpse into which attributes are providing the highest level of impact at given times, 
so that the focus (or spending) can be shifted to different attributes depending on lifecycle 
position. 
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Using the example provided earlier regarding participation in the commercial 
spare parts pool, a decision maker might agree that the impact of this benefit would 
decrease rather significantly once the commercial fleet retired. If the weight of this 
decision attribute was rather large, say 40 points out of 100 (as detennined by the Delphi 
panel of SMEs), then it would be extremely important to take this attribute into 
consideration, especially if the proposed commercial derivative acquisition was an older 
airframe and a new aircraft design was being introduced into the commercial market. 

This might indicate the commercial market was progressing toward retiring their fleet in 
favor of the newer model, leaving the USAF with an “orphan” fleet of aircraft that were 
no longer supported by commercial spare parts inventory. This is just one hypothetical 
example of the infonnation the decision tool might be able to provide. 

Decision Support Tool Design 

The input portion of the decision support tool was designed very simply by using 
a Microsoft Excel-based table fonnat. Microsoft Excel was chosen as a medium for 
constructing the tool due to its wide range of availability on government owned computer 
systems, allowing for ease of access and visibility by USAF personnel. The Excel table 
consists of the primary decision criteria or attributes for maintaining FAA certification 
(as identified in questionnaire one of the Delphi study), the average of the weights 
assigned to those attributes (taken from Delphi study questionnaire two inputs), and the 
behavior curves assigned to those same attributes (from questionnaire two). Each 
behavior curve is represented numerically by using applicable distributions (i.e. 
exponential, linear, constants, etc.). 
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Decision Support Tool Inputs 

The leftmost column of the table contains a list of attributes that were identified in 
the questionnaires. The next column consists of the average of the weights assigned to 
those attributes, followed by their impact level based upon the SME assigned behavior 
curves. For example, if the behavior curve chosen for a specific attribute was an 
exponentially decreasing curve, the impact level assigned to each lifecycle stage would 
be the nonnalized value of the curve at each lifecycle stage. Linear curves would be 
similarly depicted using linear values between 0 and 1, with a value of 1 being assigned 
the curve that showed no change. Additionally, if a nonnal curve was chosen, then a 
nonnal distribution would be used to fill in these values. 

Decision Support Tool Outputs 

An overall impact score was computed by summing the products of the attribute 
weights multiplied by the numerical values of the behavior curves. These impact scores 
are displayed at the bottom of the table and are used in bar-chart fonnat to show overall 
relative attribute impact over time. This impact score is simply meant to capture the 
shape of the behavior curve coupled with the weight of the attribute in order to provide a 
graphic depiction of the Delphi study responses. Utilizing weights to assign impact 
scores or impact grades is common in decisions with multiple attributes (Lootsma, 1999). 
Validation 

Questionnaire one consisted of statements and comments that were considered to 
be within the realm of common knowledge of the study participants, so no confusion was 
anticipated with regard to participants being able to complete it in a reasonable amount of 
time. However, since questionnaire two was slightly more complex, it was first 
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administered to another research student as a means of testing ease of understanding and 
completion time. The student was not provided any background information, but was 
simply asked to complete it by following the directions provided on the questionnaire. 
After reading the instructions for about five minutes, the student was able to complete the 
questionnaire in approximately 15 additional minutes without assistance. Based on this 
result and the fact that the actual study participants would have more background and 
experience with the material, the researcher anticipated that similar results would be seen 
when the questionnaire was administered to the actual study participants. 

Challenges 

Complex decisions such as the ones being analyzed here have some inherent 
challenges that should be highlighted. Perhaps the biggest challenge with regard to this 
study was creating a decision support tool generic enough to have a wide range of 
applicability, but not so generic that it was of little or no value to the decision maker. 
With that in mind, it is important to understand that each and every decision regarding 
military purchase of commercial derivatives and the subsequent decision of whether or 
not to maintain FAA certification is going to have attributes that are unique to the 
specific situation. Factors such as mission requirements, budget levels, and political 
direction are extremely dynamic and have a great deal of influence on USAF aircraft 
acquisitions. These types of internal and external factors are not constant, and while it 
may be possible to make point-in-time approximations, they can fluctuate dramatically 
over the course of a complex decision-making process. Because of this, the questions 
asked during the various phases of the Delphi study were purposefully broad and meant 
to solicit responses that were general in nature. It would be nearly impossible to create a 
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single decision support tool capable of providing information applicable to every possible 
MCDA acquisition scenario. 

Additional challenges came from trying to accurately identifying the costs 
associated with maintaining FAA certification. As mentioned previously, many of these 
costs would be considered sunk, fixed, or one-time costs that might lead to incorrect 
assumptions if considered equally across all FAA certification decisions. For example, 
the sunk cost associated with initial FAA certification is typically factored into the 
purchase price of the aircraft regardless if the USAF decides to maintain that 
certification. By definition, that cost is considered “sunk” because it will not be saved if 
the USAF decides not to maintain FAA certification. As such, it should not be factored 
into the decision support tool. A similar challenge arises with regard to the fixed and up¬ 
front costs associated with modifying USAF supply databases to enable tracking of FAA 
certified spare parts. These types of costs would be much higher for the first commercial 
parts-pooling endeavors, but would be relatively lower for successive MCDA 
acquisitions because the infrastructure would already be in place. Trying to incorporate 
these types of costs would likely skew decision support tool outputs. Due to the nature of 
these challenges, the focus of the tool was on potential benefits associated with 
maintaining certification, and how their impact as decision attributes changes over time. 
Summary 

The Delphi technique was well suited to the qualitative nature of this study. 
Providing clear, easy to understand questionnaires that were relatively easy to complete 
was an essential part of ensuring maximum participation, especially considering the high 
operations tempo and heavy workloads of the participants. Proper participant selection 
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was instrumental in correctly identifying the decision attributes used in the tool. Their 
strong background in USAF aircraft acquisition processes as well as front-line experience 
with FAA certified MCDA and repair processes enabled direct and meaningful feedback 
to take place and helped mitigate many of the challenges faced during the course of the 
study. Any insight gained from utilization of the decision support tool is a direct 
reflection of the valuable inputs received from each and every participant. The results 
and conclusions that were made based upon these inputs are discussed in detail in the 
next two chapters. 
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IV. Results and Analysis 


Overview 

Over the course of several months, Delphi questionnaires one and two were 
submitted to each of the participants, revised based on their inputs, and finally re¬ 
submitted once more for review in order to try and achieve consensus. After both 
questionnaires had gone through this iteration process, the revised (final) inputs were 
analyzed and used to build the decision support tool and subsequent figures, charts, and 
tables. 

Delphi Study 

Questionnaire One Results 

Based on the three purposes for questionnaire one mentioned earlier, two general 
statements regarding the risks associated with USAF purchase of commercial derivatives 
were made at the beginning of the questionnaire (Appendix B). A third general statement 
was made regarding the impact of commercial fleet size on potential FAA certification 
benefits. Statement one had two parts (1A and IB), statement two had three parts (2A, 
2B, and 2C), and the third statement had one part. Questions #1 through #3 asked each 
participant if they agreed with those statements, and if not, to provide justification for 
their disagreements. These first three questions helped establish the topic and direction 
of the questions that followed, as well as establish a baseline of knowledge to ensure all 
participants agreed on some of the basic background information regarding commercial 
derivatives and FAA certification. The final two questions (#4 and #5) on this 
questionnaire were open-ended and asked each participant to provide additional 
recommendations for decision makers considering the purchase of commercial 
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derivatives and subsequent FAA certification. These two questions were included as an 
effort to ensure no important decision criteria would be neglected or left out of the list of 
attributes for the decision support tool. 

Results obtained from the first iteration of questionnaire one were mixed, as shown 
in Table 4 and Appendix C. As mentioned earlier, although the initial participant pool 
consisted of 18 individuals, five were lost very early due to attrition, leaving a total of 13 
participants for questionnaire one. Also, since questions #4 and #5 were open ended 
questions that did not solicit an “Agree” or “Disagree” response, the columns in Table 4 
for those questions are not applicable. 


Table 4. Questionnaire One Results (First Iteration) 


Question 

Agree in Full or Agree 
with Comment 

Disagree 

1A 

10 

3 

IB 

10 

3 

2A 

11 

2 

2B 

7 

6 

2C 

11 

2 

3 

6 

7 

4 

N/A 

N/A 

5 

N/A 

N/A 


Based on the mixed response received on statements 2A and 3, both statements 
were modified to incorporate additional participant comments. The modified statements 
can be seen in Appendix D. Since an overwhelming majority of participants agreed with 
the remaining statements, they were left unmodified and not submitted as part of the 
second iteration. By only re-submitting modified statements in the second iteration, 
participants did not have to spend time re-reading information they had already 
commented on and impact on participant workload was minimized. The two modified 
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statements were resubmitted to all participants, resulting in a much higher level of 
consensus, as shown in Table 5. All additional comments were carefully considered to 
detennine if any further modification was needed. 


Table 5. Questionnaire One Results (Second Iteration) 


Question 

Agree 

Disagree or Agree with 
Comment 

1A 

10 

3 

IB 

10 

3 

2A 

11 

2 

2B (Modified) 

13 

0 

2C 

11 

2 

3 (Modified) 

12 

1 

4 

N/A 

N/A 

5 

N/A 

N/A 


The open-ended inputs received from questions #4 and #5 were compiled and then 
used to build upon and confirm preliminary research that had been accomplished in the 
area of FAA certification decision criteria. 

Due to the nature of the costs associated with FAA certification that was discussed 
previously, decision criteria were primarily focused on the potential benefits of 
maintaining FAA certification. The decision criteria (listed as “attributes” in the decision 
support tool) are identified below, in no particular order, using the same verbiage and 
sequence that was used on the questionnaire. This list contains the ten attributes that 
have the most impact on the decision to maintain FAA certification, as detennined by the 
Delphi study participants. When given an opportunity to add more attributes to this list, 
none of the participants did so, which may indicate that the list captures a majority of the 
benefits associated with maintaining FAA certification. 
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Decision Criteria (or “Attributes”) List 

1. Participation in the commercial parts (spares) pool 

2. Access to established commercial maintenance support and infrastructure 

3. Access to an established commercial supply chain 

4. Use of existing manuals, repair/failure data, and technical data 

5. Use of existing FAA approved processes (Service Bulletin and Airworthiness 
Directive processes, repair processes, etc.) 

6. Improved control and tracking of aircraft configuration data 

7. Ability to participate in cross-servicing with European Aviation Safety Agency 
(EASA) and FAA fleets 

8. Ability to sell demilitarized MCDA back to commercial market 

9. Reduced need to acquire proprietary OEM design data or flight test data 

10. Improved safety due to increased FAA oversight 

With the vast majority of participants now in agreement with the statements on the 
first questionnaire, as well as all decision criteria properly identified and all inputs 
compiled regarding additional recommendations for decision makers, round one of 
questioning was considered complete. A complete record of all inputs can be found in 
Appendix C. 

Questionnaire Two Results 

Questionnaire two (Appendix E) was divided into two parts based on the purposes 
outlined for this questionnaire that were mentioned earlier. The first part consisted of a 
weight identification chart for the decision criteria (from questionnaire one). This simple 
chart listed the primary FAA certification decision criteria (10 in all) in the left column 
with a blank “weight score” column on the right. Participants were asked to distribute 
100 points total across all 10 criteria. Weight scores and averages can be found in 
Appendix F, however Figure 4 is a pie chart depiction of those averages (as a percentage 
of the total), with abbreviated attribute titles listed by order of weight (highest to lowest) 
on the right of the figure. As can be seen in Figure 4, the top three attributes combined 
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weights, as assigned by the SMEs, represent nearly half of the 100 points allotted. 
Additionally, the top five attributes were assigned nearly three-quarters of the total 
weight allotment. This distribution of weights helps to confirm that a simple rank-order 
of attributes (i.e. 1 through 10) would not have sufficiently captured attribute impact. 
Instead, the top five attributes are considered to have disproportionately more impact on 
the decision to maintain FAA certification than the bottom five attributes. 


Attribute Weight Distribution 



■ Parts Pooling 

■ Access to SC Infrastructure 

■ Use of Existing Data 

■ Use of FAA Processes 

■ Access to Proprietary Data 

■ Improved Safety/Quality 

■ Maintenance Support 

■ Commercial Sell Back 
Improved Config Control 

■ FAA/EASA Cross Servicing 


Figure 4. Attribute Weight Pie Chart 


Across most of the participants, weights assigned were very similar for each 
attribute, and in fact, most weight values fell within a 10-point range with only a few 
outliers. The presence of outliers is likely due to varying interpretations of the attribute 
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definitions. This variation is one of the risks associated with using fairly ambiguous or 
qualitative data. Overall, two attributes had all values fall within a 10-point range, four 
attributes had a single outlier, three attributes had two outliers, and one attribute had three 
outliers, as depicted in Table 6. The individual values of these outliers are also shown in 
the table. 


Table 6. Attribute Weight Range and Number of Outliers 


Attribute 

Assigned Weight 
Range 

Number of Outliers 

Outlier Values 

Parts Pooling 

15-25 

3 

5,35,40 

Access to SC 
Infrastructure 

10-20 

0 

N/A 

Use of Existing Data 

5-15 

1 

30 

Use of FAA Processes 

5-15 

1 

30 

Access to Proprietary 
Data 

0-10 

2 

25,30 

Improved Safety/Quality 

0-10 

2 

15,25 

Maintenance Support 

0-10 

0 

N/A 

Commercial Sell Back 

0-10 

1 

25 

Improved Configuration 
Control 

0-10 

2 

12, 15 

FAA/EASA Cross 
Servicing 

0-10 

1 

15 


The second part of the questionnaire consisted of a fairly complex scheme of 
figures and questions designed to help determine individual attribute behavior over time. 
This section asked participants to select from an assortment of attribute behavior curves 
that they felt best identified the correct behavior for each attribute over time. Two points 
were included on the non-linear curves to allow participants to assign transition points 
based on the lifecycle timeline provided (linear curves have no distinct transitions, and 
thus no need for the transition points). With 10 attributes for example, 10 curves and 10 
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sets of points (linear curves excluded) were selected representing fleet lifecycle stages. 
Despite the relative complexity of this questionnaire, it appeared that the study 
participants understood the meaning behind the curves, since their selections anecdotally 
appeared consistent with what would be considered intuitive with regard to the attributes. 
For example, the attributes that would be dependent on an active commercial fleet were 
assigned curves that displayed diminishing levels of impact as the commercial fleet 
entered retirement. As with the first questionnaire, results were mixed, but some 
common themes could be seen, especially among some of the top weighted attributes. 
When the results of questionnaire two were compiled and re-submitted to the participants 
in the second iteration (in an attempt to improve concurrence levels), none of the 
participants made any changes to their answers, and thus questionnaire two was 
considered to be complete. 

With 10 inputs received from this second questionnaire, focus was first given to 
attributes in which at least half the participants agreed on a behavior curve. Interestingly, 
this was precisely the case with five of the six top weighted attributes. In other words, 
when ranked by weight, attributes 1, 2, 3, 4, and 6 (as identified in Figure 4) had behavior 
curves that were agreed upon by at least half of the Delphi study participants. 

Considering that there were a total of nine curves to choose from, this was detennined to 
represent a fairly strong concurrence. Additionally, the lifecycle timeline points chosen 
(from six available) were also very closely matched among these same attributes. These 
results can be seen in Appendices G, H, and I, but a summary of these specific five 
attributes (listed by highest to lowest average weight) can be seen in Table 7. As a 
reminder, linear behavior curves (behavior curves #7, #8, and #9 on questionnaire two) 
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did not require selection of lifecycle timeline points, and as such, do not have values 
listed in those table columns. 


Table 7. Attribute Behavior Curve Comparison (Minimum 50% Concurrence) 


Attribute 
(abbreviated title) 

Behavior Curve 
Shape (chosen by 
at least half of 
participants) 

Behavior Curve 
Point A (chosen by 
at least half of 
participants) 

Behavior Curve 
Point B (chosen by 
at least half of 
participants) 

Parts Pooling 

4 

2 

5 

Access to SC 
Infrastructure 

4 

2 

5 

Use of Existing 
Data 

8 

N/A 

N/A 

Use of FAA 
Processes 

8 

N/A 

N/A 

Improved 

Safety/Quality 

8 

N/A 

N/A 


For the “parts pooling” and “access to supply chain (SC) infrastructure” attributes 
shown in Table 7, behavior curve #4 was an exponentially decreasing curve. Point 2 on 
the lifecycle timeline represented the mid-production stage and point 5 represented the 
stage at which the commercial fleet began to retire. Curve #4 is shown in Figure 5 
below. 



Figure 5. Behavior Curve #4 from Questionnaire Two 
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For these two criteria, it can be seen in Figure 5 that at least half of the participants 
felt that the benefits associated with the “parts pooling” and “access to SC infrastructure” 
attributes were fairly constant from mid production until the beginning of commercial 
fleet retirement, but then decreased dramatically. 

For the remaining three criteria in Table 7, behavior curve #8 was chosen. Curve 
#8 (Figure 6) is simply a flat line that depicts no change over time. In other words, most 
participants agreed that the benefits associated with these three attributes were fairly 
constant over the lifecycle of the fleet. Note that there are no transition points in curve #8 
because the rate is fairly constant through all lifecycle timeline points. Appendix L 
shows a graphic depiction of how these five attributes behave over time. 



Fairly constant rate through all 
points (little to no change) 

Figure 6. Behavior Curve #8 from Questionnaire Two 

Behavior curve selection for the remaining attributes was much more mixed, and 
while a majority concurrence was not achieved, there were curves that were selected 
more often than others. For example, the attribute that was weighted the fifth highest, 
“access to proprietary data,” had behavior curve #4 chosen more often than any other 
curve. Additionally, Point A was identified as “early production” by in all cases that 
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curve #4 was chosen. Point B was identified as “begin MCDA retirement” most often by 
those that chose curve #4. Results for the remaining attributes are shown in Table 8, and 
complete results for all attributes can be found in Appendices G, H, and I. It is important 
to note that the transition points (A and B) identified in Table 8 are not necessarily the 
ones that were chosen most often among all participants, however they were the ones 
chosen most often with the associated curve shown in the table. 

In order to provide a slightly different perspective, the responses to questionnaire 
two were divided into two groups based on the positions held by the participants at the 
time. One group included responses from participants whose jobs were primarily related 
to the field of acquisitions, while the second group included responses from participants 
whose jobs were primarily sustainment related. Interestingly, even after separating the 
groups in this manner, the top three attributes (as ranked by assigned weight) for both 
groups were the same as before the separation: “parts pooling,” “access to supply chain 
infrastructure,” and “use of existing data.” However, with the sustainment related group, 
“access to proprietary data” and “commercial sell back” replaced “use of FAA processes” 
and “access to maintenance support” as attributes ranked fourth and fifth. With the 
acquisitions related group, “access to maintenance support” replaced “improved 
safety/quality” as the fifth attribute. Graphic depiction of these results can be seen in 
Appendices M (sustainment) and N (acquisitions). Of note in Appendix M is the 
relatively short window of impact for the “commercial sell back” attribute, due to the 
associated behavior curve. It is also important to note that these two additional graphs 
were only created to compare the differences in weight assigned by the two groups, and 
that the curves used to create the graphs were from Tables 7 and 8, as applicable. 
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Table 8. Attribute Behavior Curve Comparison (Non-Majority Concurrence) 


Attribute 
(abbreviated title) 

Behavior Curve 
Shape (chosen 
most often by 
participants) 

Behavior Curve 
Point A (chosen 
most often with 
associated curve) 

Behavior Curve 
Point B (chosen 
most often with 
associated curve) 

Access to 
Proprietary Data 

4 

1 

6 

Maintenance 

Support 

4 

2 

5 

Commercial Sell 
Back 

6 

4 

5 

Improved 

Configuration 

Control 

8 

N/A 

N/A 

FAA/EASA Cross 
Servicing 

8 

N/A 

N/A 


Decision Support Tool 

As shown in the Decision Support Tool in Appendix J, attributes that were 
identified as having their behavior over time associated with curve #8 (little to no change) 
were given a constant impact score of 1 for all lifecycle stages. There were five attributes 
in this category. Four of the remaining attributes were identified as having their behavior 
over time associated with curve #4. This was an exponentially decreasing curve that was 
fairly constant between the two transition points (A and B) followed by a dramatic 
decrease. In three of these cases, points A and B were identified to be “mid production” 
and “commercial retirement” respectively. As such, an exponential distribution was used 
to determine the impact level values for the first five lifecycle stages, dropping off to zero 
for stage 6. In the fourth case where curve #4 was chosen, transition points A and B were 
identified to be “early production” and “begin MCDA retirement” respectively. In this 
case, an exponential distribution was used to determine impact level values across all six 
lifecycle stages. In other words, due to the different choice for point B in this case, 
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impact level value did not drop off to zero, as in the first three cases of curve #4. For the 
final attribute, curve #6 was chosen which depicts exponential decrease between points A 
and B. In this case, point A was identified to be “post production” and point B was 
identified to be “begin commercial retirement.” Due to these late-cycle selections for 
transition points, impact levels prior to “post production” were set at zero, followed by 
exponentially decreasing values for the remainder of the timeline. All of these results are 
more clearly identified in Appendices G, H, and I. 

Lifecycle Impact Score 

By computing the product of impact level and attribute weight values, an impact 
score was obtained for each attribute at every stage of the lifecycle timeline. In other 
words, with 10 attributes and six lifecycle stages, a total of 60 impact scores were 
obtained. Impact scores for each lifecycle stage were summed, resulting in a value that 
represents the combined impact of all attributes at each given stage. A summary of these 
values is shown in Table 9. 


Table 9. Combined Lifecycle Impact Scores 


Lifecycle 

Stage 

Early 

Production 

Mid 

Production 

Late 

Production 

Post 

Production 

Begin 

Commercial 

Retirement 

Begin 

MCDA 

Retirement 

Impact 

Score 

92.97 

92.45 

91.07 

93.54 

77.18 

47.20 


Attribute Impact Lifecycle Behavior 

As shown in Appendix K, these combined impact scores dip slightly from early to 
mid-production, and then again from mid to late production. They then peak during post 
production, followed by a dramatic decline at the beginning of commercial fleet 
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retirement. This dramatic decline continues through the beginning of MCDA retirement. 
These values were used to create a graphic depiction of attribute impact relative to the 
lifecycle timeline, as shown in the chart in Appendix K. 

Summary 

There are many challenges associated with achieving majority consensus over the 
course of a fairly short Delphi study, and these challenges become even more apparent 
when the data collected is qualitative in nature. Overall, two rounds (or questionnaires) 
were deemed sufficient to gather the necessary infonnation to construct and populate the 
decision support tool. Considering the participant attrition that was already occurring 
after two rounds, a third round of questions would likely have not provided a significant 
amount of additional information. Additionally, varying interpretations and opinions can 
often lead to a wide range of differing responses, especially when the study participants 
have extremely diverse backgrounds, as in this study. Despite these challenges and 
diversification, however, many responses to questionnaires one and two were surprisingly 
similar. For example, the final list of attributes used to create the decision support tool 
was agreed upon by all participants. The weight scores assigned to these attributes were 
also very close. While the selection of behavior curves and transition points varied 
greatly for some attributes, they were fairly similar for the attributes with the highest 
weight. This similarity allowed for some simple but useful conclusions to be drawn from 
the data, discussed further in the next chapter. 
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V. Discussion 


Overview 

As stated in Chapter 1, the primary objective of this research was to helps identify 
criteria that USAF leaders and decision makers could use to detennine whether 
maintaining FAA certification of MCDA was beneficial when compared to maintaining 
only military certification. In order to fulfill this objective, three research questions had 
to be answered. 

Question 1: What are the decision criteria USAF leaders should consider when 
determining whether or not to maintain FAA certification of MCDA? 

Question 2: How do the costs and benefits associated with maintaining FAA 
certification of MCDA change over time? 

Question 3: If the choice is made to maintain FAA certification of MCDA, how 
long should it be maintained? 

Question 1 was answered through the participant responses received from Delphi 
questionnaires one and two that validated a list of 10 decision criteria for decision makers 
to consider when detennining whether or not to maintain FAA certification. These 
criteria represented the attributes for the MADM support tool described in Chapter 3 and 
seen in Appendix J. As outlined in Chapter 2, the focus of the attributes was on the 
potential benefits of maintaining FAA certification, since costs associated with FAA 
certification were primarily sunk, fixed, or one-time (up-front) expenses. Question 2 was 
also answered through participant responses received from Delphi questionnaire two, 
where attributes were assigned a weighted value and then matched to behavior curves 
that identified changes over time. This data represented the second set of inputs to the 
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MADM support tool. Finally, the output of the MADM support tool (Appendices K and 
L) provided the answer to Question 3 by displaying graphic representations of attribute 
impact over the typical lifecycle of a fleet of aircraft. These graphs allow decision 
makers to not only identify which attributes have the most impact, but at what point 
during the lifecycle timeline impact levels diminish. 

Conclusions 

Looking solely at the weight values assigned to the 10 attributes, decision makers 
can conclude that most of the SMEs that participated in this study feel that participation 
in a commercial parts (spares) pool and access to a well-established commercial supply 
chain are the two attributes that have the most impact on the decision to maintain FAA 
certification. In fact, these two attributes alone account for more than 35% of the total 
weight of all attributes. However, weight value should not be considered without also 
taking into account the diminishing nature of these attributes, as indicated by their 
behavior over time. In this case, impact of these two attributes diminishes rapidly as the 
commercial fleet enters the retirement stage of the fleet lifecycle. Take for example, a 
MCDA purchased near the end of production. Consider also that the decision maker is 
considering maintaining FAA certification of that MCDA. The decision maker is also 
considering allocating a portion of the budget to pay the up-front costs associated with 
upgrading military supply databases and spare parts tracking regulations in order to take 
advantage of commercial parts pooling opportunities. By analyzing the output of the 
decision support tool, the decision maker can see that there is a fairly short window of 
opportunity to take advantage of the benefits associated with that decision. The decision 
maker can also see that as the commercial fleet begins to retire, a very large portion of 
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those benefits diminish rapidly, which greatly reduces his or her return on investment. 

The attribute associated with access to proprietary OEM design data or flight test data 
behaves in a similar diminishing fashion, however not as quickly. 

In another scenario, the MCDA in question might be in the mid-production stage 
leading the decision maker to believe that the cost of maintaining FAA certification will 
be offset by the benefit associated with access to established commercial maintenance 
support and infrastructure. While it is true, according to SMEs in this study, that such 
access is beneficial, this particular attribute was assigned less than 7% of the total weight, 
as compared to the other attributes. If access to maintenance support is the primary 
decision criteria being considered, then the cost of maintaining FAA certification may not 
be justified in this case. On the other hand, if use of existing manuals and repair data or 
use of existing FAA processes (such as service bulletin/airworthiness directive and repair 
processes) are the primary decision criteria being considered, the decision maker can see 
that these two attributes not only represent nearly 24% of assigned weight, but also that 
their impact level remains fairly steady over time. In other words, according to the SMEs 
that participated in this study, the benefits associated with these two attributes do not 
depend on what stage of the lifecycle timeline the particular MCDA happens to be in. 

A final scenario involves the attribute associated with selling the MCDA back to the 
commercial market. Maintaining FAA certification of the MCDA throughout its military 
lifecycle makes this a much simpler process due to ease of records transfer, maintenance 
tracking requirements, etc. However the MADM support tool shows a very short 
window of opportunity for this benefit coupled with a low weight score. While still a 
valid benefit, it does not ra nk very high when compared to other benefits. The impact 
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levels of the remaining benefits were determined to remain fairly constant over time by 
varying degrees of consensus of the Delphi participants (as shown in Appendix G). 

These remaining attributes also represent the lower end of the weight scale. 

Assumptions and Limitations 

While the overall goal of this research is to identify potential benefits that can be 
achieved through USAF collaboration with the FAA and commercial industry, some of 
this research also focuses on potential improvements that can be made. As such, the level 
of impact of some of the potential benefits outlined is dependent on individual 
circumstances. For example, it is assumed that if the criteria are met for the USAF to 
access the commercial parts pool for a particular MCDA, the commercial industry will 
allow that access. In other words, maintaining FAA certification does not in and of itself 
guarantee the benefits associated with that certification. Agreement in these areas must 
be obtained from all parties involved. Additionally, it is beyond the scope of this 
research to provide an all-inclusive decision making tool for all situations regarding FAA 
and military certifications, but certainly there are specific costs and benefits that are 
analyzed and explained that will help decision makers choose the appropriate course of 
action. The infonnation provided might be used as a heuristic that, when combined with 
other applicable information, can provide further insight into the consequences of the 
certification choices, while also acting as a guide to a more accurate and optimal course 
of action. 

There are numerous challenges inherent in trying to simplify decisions with many 
variables. In this case, the variables that make up decision criteria involved in 
detennining whether or not to maintain FAA certification of MCDA are both dynamic 
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and somewhat ambiguous, and as such are hard to quantify. Extensive research on 
exactly how the commercial market can support DOD logistics is not readily available, 
and successful examples of FAA certified MCDA have not been thoroughly captured and 
analyzed in terms of real costs and benefits. In fact, most DOD aircraft drop the FAA 
certification after military certification is achieved, leaving the military customer with an 
“orphan fleet.” 

Additionally, due to short timelines, high operations tempo, or manning 
constraints, there is typically not enough time to conduct an adequate amount of market 
research prior to new aircraft acquisitions. Other factors that may negatively affect a 
decision maker ability to choose the most optimal solution include 50-50 requirements as 
outlined in Title 10 United States Code Section 2464, which requires 50 percent of depot 
operations and facilities to be government owned and operated by government personnel. 
While this reduces some of the risk associated with the government having too much 
dependence on the commercial market, it may prevent decision makers from being able 
to take advantage of existing commercial resources. While this type of restriction may 
have been necessary at one time, some would argue that it is no longer justified, given the 
current market’s flexibility and response capabilities (Stockman et ah, 2011). 

Other challenges emerge when military capabilities are declared “core,” meaning 
that they are essential to military operations, and as such must be supported by military 
infrastructure and personnel. For example, the USAF latest MCDA acquisition, the KC- 
46 tanker (commercial derivative of B767), may be declared “core” despite existing 
commercial infrastructure and support. Such a decision would drastically increase 
overall lifecycle sustainment costs. Another example can be seen with the USAF 
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acquisition of the C-27J, which was declared core despite a business case analysis that 
stated CLS would be more cost effective for sustainment. During the early stages of 
these types of acquisitions, requirements to comply with Federal Acquisition Regulations 
(FAR) also leads to increased administrative costs that may be unnecessary. 

Recommendations 

It is recommended that the MADM decision support tool presented in this research 
be incorporated as part of the FAA certification decision making process. This tool is not 
meant to represent an all-inclusive guide to detennine the appropriate response to such 
decisions. It is meant to be used as a reference to achieve an additional level of 
understanding into FAA certification decision criteria and how they behave over time. It 
is also recommended that decision makers review the complete responses provided to 
both Delphi study questionnaires, as they may provide additional insight or help guide the 
decision maker to a more optimal solution. The complete record of questionnaire 
responses can be found in Appendices C, D, G, H, and I. 

Areas for Future Research 

Due to the high amount of weight assigned to the two attributes associated with 
access to commercial parts and the commercial supply chain, a potentially beneficial area 
for future research would be to take a closer look at the costs associated with upgrading 
military systems to take advantage of that access. The relatively short timeline of this 
research did not allow for adequate investigation into this area. By obtaining a rough 
estimate of this cost, future decision makers may be able to justify allocating a portion of 
the budget to this purpose. It is possible that the cost may be too great to be offset by an 
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existing MCDA fleet, but instead would be offset over the course of several future 
MCDA purchases. 

An additional area for future research would be to expand this topic to include all 
branches of military service. For example, the U.S. Navy also has MCDA currently in 
use and valuable insight would be gained by including Navy personnel as participants in 
the Delphi study. Including all military branches would facilitate the sharing of lessons 
learned and best practices. Finally, as discussed in Chapter 1, it may also be beneficial to 
also look into streamlining the certification process itself. The process has evolved into a 
very complex set of procedures and requirements and might become more efficient if 
simplified. It may also be beneficial to look at the Military Type Certification process 
that occurs when FAA certification is not maintained. 

Summary 

The decision regarding whether or not to maintain FAA certification of MCDA is 
certainly not an easy one to make. Decision makers may not always have a complete 
picture that accurately outlines the costs and benefits of their choices. Heavy workloads 
coupled with a high operational tempo can detract from the time needed for adequate 
research, and when coupled with a tight timeline, the situation can lead to a less than 
optimal outcome. Additional challenges are presented when the infonnation that is 
available is vague or difficult to interpret, depending on one’s background and 
experience. However, the potential benefits of collaboration in the area of certification 
can be substantial, particularly as we continue to build a more global economy and 
increase the need to collaborate on an international level. Only when decision makers 
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have all the information they need to identify these cooperative opportunities will they be 
in a position to take full advantage of them. 

The MADM support tool developed as part of this research coupled with the 
participant responses to the Delphi questionnaires provides the answers to the original 
research questions outlined in Chapter 1. While somewhat generic in nature, the MADM 
support tool nonetheless provides further insight into the decision criteria and may be 
used as part of an overall decision making strategy. 
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Appendix A. List of Acronyms 


CDA - Commercial Derivative Aircraft 
CLS - Contractor Logistics Support 

CLSSA - Cooperative Logistics Supply Support Arrangement 

CNS/ATM - Communication, Navigation, Surveillance and Air Traffic Management 

COTS - Commercial-Off-the-Shelf 

CRS - Congressional Research Service 

DOD - Department of Defense 

DOT - Department of Transportation 

FAA - Federal Aviation Administration 

FAR - Federal Acquisition Regulations 

FMS - Foreign Military Sales 

ICAO - International Civil Aviation Organization 

INAS - International Airspace System 

MADM - Multi Attribute Decision Making 

MCA - Military Certificate of Airworthiness 

MCDA - Military Commercial Derivative Aircraft 

MCO - Military Certification Office 

MOA - Memorandum of Agreement 

MTC - Military Type Certificate 

OEM - Original Equipment Manufacturer 

SME - Subject Matter Expert 

TAA - Technical Airworthiness Authority 

TSOA - Technical Standard Order Authorization 

USAF - United States Air Force 
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Appendix B: Delphi Study Questionnaire One 


Purpose: 

The United States Air Force continually searches for ways to conserve materials and costs 
associated with our mission. A large portion of these costs are attributed to the acquisition of 
new aircraft. Air Force leaders can potentially reduce acquisition costs if they are able to take 
advantage of some of the benefits associated with utilizing aircraft that have already been 
developed by the commercial industry (commercial derivative aircraft or CDA). One potential 
benefit of utilizing CDA is related to the certification of the aircraft and its components by the 
Federal Aviation Administration (FAA). Currently, research is being conducted to help Air Force 
leaders determine if maintaining or achieving FAA certification of CDA or their associated 
repair processes is beneficial when compared to military certifications. 

Participation: 

Your participation in this study is COMPLETELY VOLUNTARY. This research is to be 
conducted by Captain Mike Low in fulfillment of a Master’s degree program at the Air Force 
Institute of Technology (AFIT). This work is sponsored by the Air Force Materiel Command’s 
Logistics Directorate. 

Confidentiality: 

This study will not collect personal identifiers or specific demographic infonnation. Due to the 
nature of the electronic mail data collection process, the other study participants will know who 
you are but they will not know which comments are yours. At the end of the research, individual 
participant inputs will not be disclosed. You may withdraw from this study at any time without 
penalty; if you do, your data will not be used in the research. Your decision to participate or 
withdraw will not jeopardize your relationship with your department, the Air Force Institute of 
Technology, the Air Force, or the Department of Defense 

Instructions: 

You will be emailed a knowledge based questionnaire that will ask you to review a short set of 
criteria that Air Force leaders would potentially use to make decisions regarding the use of 
commercial derivative aircraft and certification processes. After reviewing the criteria, you will 
be asked to provide comments and recommendations regarding the validity of those criteria 
based on your current knowledge and experience. This study is simply a method of validating the 
criteria listed, so all comments are welcome, and participant names will NOT be used in the final 
project. 

Over the course of the study, you will receive a series of emails (approximately 3-5 total). The 
first email will ask you to review the criteria and provide any comments or corrections you might 
have. Comments will be incorporated into the criteria and then resubmitted to participants until 
consensus is achieved. Please base all of your responses on your current level of knowledge and 
experience and make your comments as clear and concise as possible. With regard to the 
questions that will be asked, there is no “right” answer, so any comments you provide will be 
extremely helpful. 
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Contact Info: 

If you have any questions, comments, or concerns about this study, please contact Capt Mike 
Low using the infonnation below. 

AFIT/ENV BLDG 641/Room 201 
2950 Hobson Way 

Wright Patterson AFB OH 45433-7765 
Email: Michael.low@afit.edu 
Advisor: Alan.johnson@afit.edu 

Phone: DSN 785-3636 x4703, commercial (937) 255-3636 x4703 
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Please review the following three statements below. Then, based on your current knowledge and 
experience, answer the five questions at the bottom of the page. 

1. Regarding Air Force purchase of commercial derivative aircraft, the risk associated with 
purchasing aircraft near the end of the planned production run is due to primarily to the 
following factors: 

a. The length of time that a commercial supply chain infrastructure will remain in 
place is shorter when compared to purchases made at the beginning of the 
production run. 

b. The length of time that a systems engineering support will be available is shorter 
when compared to purchases made at the beginning of the production run. 

2. Regarding Air Force purchase of commercial derivative aircraft, the risk associated with 
purchasing aircraft near the beginning of the planned production run is due to primarily 
to the following factors: 

a. The cost of each airframe is higher at the beginning of production when compared 
to later stages (due to production efficiencies achieved over time, manufacturer 
pricing changes, etc.) 

b. The level of quality of aircraft purchased is lower during initial stages of 
production when compared to later stages (due to employee experience gained 
over time, lessons learned, etc.) 

c. The production schedule is less stable at early stages of production and aircraft 
are less likely to be delivered according to schedule when compared to later 
stages (due to manufacturer scheduling and production experience increases, etc.) 

3. Many of the benefits associated with maintaining or pursuing FAA certification of Air 
Force commercial derivative aircraft or repair processes can only be achieved when there 
is a large commercial fleet still in use by the commercial industry. 

Question 1. Considering statement 1 above, would you say items la and lb accurately identify 
the primary risks associated with Air Force purchases of commercial derivative aircraft made at 
the end of the scheduled production run? If not, please identify items you would add or remove. 

Question 2. Considering statement 2 above, would you say items 2a, 2b, and 2c accurately 
identify the primary risks associated with Air Force purchases of commercial derivative aircraft 
made at the beginning of the planned production run? If not, please identify items you would add 
or remove. 

Question 3. Considering statement 3 above, would you agree with the statement made regarding 
the benefits of FAA certification? If not, please identify why you disagree. 

Question 4: Considering decisions Air Force leaders must make regarding whether or not to 
utilize commercial derivative aircraft, what additional comments or recommendations would you 
provide? 

Question 5: Considering decisions Air Force leaders must make regarding whether or not to 
maintain or pursue FAA certification of commercial derivative aircraft and their associated repair 
processes, what additional comments or recommendations would you provide? 
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Appendix C: Delphi Study Questionnaire One Responses 


Question 1 Responses. 

IA. True. Using the KC-46 as example, the end of the production run is 2028. The AF will 
have 179 aircraft with projected life span of 40-50 yrs. Major commercial carriers will have 
replaced their 767 with newer types of aircraft by that time. Only the minor carriers, smaller 
foreign carriers, and cargo haulers will likely be using the 767 after 2025-2030. Most major 
supply chain vendors will stop supplying for the 767 due to diminishing market and need from 
the commercial side. The AF requirement for commercial supply support for 179 aircraft, while 
important to the AF, is a small requirement in the commercial world. 

IB. Basically true. The AF will have to rely more on its own AF engineering and OEM 
engineering contract for long tenn support, similar to KC135 program. 

Risk at the end of a production run may be associated with the supply and service infrastructure, 
if the production run is relatively small. However, for aircraft with a large commercial 
production run (e.g., B737 family of aircraft), I think this is statement does not apply as much. 
See answer to question 4 below. I feel statement lb is not accurate. Regarding systems 
engineering support, FAA regulations (14 CFR 21 and 14 CFR 26) require that for aircraft that 
remain in service, the holder of a design approval (type certificate, amended type certificate, 
supplemental type certificate; e.g., Boeing) must supply the infonnation necessary (not 
necessarily for free) for aircraft operators to operate and maintain those aircraft in an airworthy 
condition. Case in point is the Beech Starship. Only 53 aircraft were produced following 
certification due to various reasons. Beech determined that it would be cost prohibitive to 
support such a small fleet size and began to scrap and destroy the aircraft under its control. 

Beech attempted to acquire all of the privately owned aircraft, but was unsuccessful. Therefore, 
Beech is required to maintain and provide the instructions for continued airworthiness for this 
aircraft, (only 5 or 6 aircraft still in service) The design approval holder must also 
provide support for the continued airworthiness of and safety improvements for transport 
category airplanes with regard to electrical wiring interconnection systems maintenance, fuel 
ta nk flammability and damage tolerance data for repairs and alterations. This support may 
include performing assessments, developing design changes, developing revisions to Instructions 
for Continued Airworthiness (ICA), and making necessary documentation available to affected 
persons. Therefore, as long as an aircraft remains in service, some sort of systems engineering 
support should be available to a military operator of that aircraft. 

Yes, la and lb are accurate statements. Another factor to consider is the number of commercial 
aircraft in active service. For a highly successful aircraft, it is likely that supply chain and 
engineering support will last much longer than with an aircraft with low numbers in active 
commercial use. 

Yes. 

Yes, I agree with those statements. 

I agree both statements la and lbare correct. 

Yes, I agree with la and lb. 
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The availability of a supply chain is more dependent on the technology being used rather than 
where in the production life cycle an airplane is. Out of production aircraft still have up to a 20 
year life cycle left in the commercial environment with both second and third tier airlines as well 
as cargo carriers. Engineering support is always available for a fee and isn’t related to where in a 
production run an aircraft is. As long as there is an airplane model flying the owner of the type 
design is responsible for supporting it. 

The response assumes production run is limited to an optimal run with profit maximization for 
the corporate entity. I would not agree with the position though I agree both your statements hold 
true. I would assert that the main issue with the late or end production run purchases of 
derivative aircraft is technical data cost and availability. This would include aircraft 
specification data and trend data for extension of Reliability and Maintainability (R&M) effort. 
Far too often the end of production run is assumed programmed improvement will no longer 
occur. Supply chain can be mitigated through Diminishing Manufacturing Sources (DMS) 
programs currently in place. Additionally, government engineering has the ability to pick up any 
required engineering actions, provided the data is made available. 

IA. True. Using the KC-46 as example, the end of the production run is 2028. The AF will 
have 179 aircraft with projected life span of 40-50 yrs. Major commercial carriers will have 
replaced their 767 with newer types of aircraft by that time. Only the minor carriers, smaller 
foreign carriers, and cargo haulers will likely be using the 767 after 2025-2030. Most major 
supply chain vendors will stop supplying for the 767 due to diminishing market and need from 
the commercial side. The AF requirement for commercial supply support for 179 aircraft, while 
important to the AF is a small requirement in the commercial world. 

IB. Basically true. The AF will have to rely more on its own AF engineering and OEM 
engineering contract for long tenn support, similar to KC135 program. 

If you are inferring that part availability is a risk driver due to potential shortage of parts 
delivered near the end of a production run in a commercial environment then I would have to 
disagree. Part quality in an FAA environment is higher than our military variants and response 
time for a production run is quicker in the commercial world than our defense OEM’s. Systems 
Engineering support does ramp up in the beginning of a production run and drop off over time as 
the product matures and more emphasis is placed on post-delivery support (for warranty) but I 
have not seen examples of where this is a driving risk factor for purchasing aircraft near the end 
of a production run . The risks that we have seen with purchasing near the end of a production 
run have to do with procuring a common configuration across the fleet you are ordering, and the 
stability of aircraft pricing and lead-time based on the amount of parts in the supplier system to 
support your order. 

1 A. Disagree: Although this may pose a slight risk, the supply chain infrastructure for 
commercial derivative aircraft usually remains robust due to the number of suppliers competing 
to provide materials and services. If the OEM (Original Equipment Manufacturer) were the only 
source for materials and services, this statement would be true only of the OERM decided not to 
support their product.. 

IB. Disagree: OEM’s will invariable provide engineering support for a product beyond its 
production run. Additionally, Operators use internally engineering staff as well as DER 
(Designated Engineering Representatives) to provide engineering support rather than OEM 
engineering, mainly due to the fact the OEM Engineering and be cost prohibitive in the 
commercial world. 

Agree. 
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Question 2 Responses. 

I believe 2a and 2c to be fairly accurate. You may want to add another qualifier to item 2c. 

Since commercial production is highly market dependent, market volatility could adversely 
impact production, especially when the military is trying to piggy-back on a commercial 
production run for a small number of aircraft. Item 2b is not accurate. Production quality (at 
least with regard of airworthiness or safety of flight) is strictly controlled and regulated by the 
FAA; therefore, the level of quality of the first aircraft off of the production line should not be 
any less than that of the last production unit. 

Yes, I would say the statements are generally true, risks are higher for a new production aircraft. 
Yes. 

I agree with 2a and 2c. I do not agree with 2b, reason being that level of quality is required to be 
maintained throughout the production stages. There are quality requirements for obtaining and 
maintaining FAA certification. 

I agree 2a, 2b, 2c are correct. 

Yes, I agree with 2a, 2b and 2c. 

2A. Disagree. Buying a CDA that has been in production for a number of years should reduce 
the cost because all of the engineering and production line bugs have typically been resolved. 
While you always have continuous process improvement it doesn’t provide huge gains in 
efficiencies for a mature production line. Manufacturing pricing will typically be lower in the 
beginning because of a "bulk" buy upfront and competition. Also pricing may increase in later 
years because of diminishing parts availability from commercial vendors as the commercial fleet 
downsizes. 

2B. Disagree. For typical CDA production the production line is mature as the AF is procuring 
a proven CDA, not a new design. Most of the workforce is already experienced on the specific 
CDA line with proven training programs for new hires. While there is always room for 
improvement with any production line, the quick and easy "low hanging fruit" has already been 
picked when the line was new and immature. Process improvements to a mature line are harder 
to implement and likely do not carry the same dramatic levels of improvement. 

2C. Disagree. Typical CDA the AF procures are already being produced and mature. Thus 
reduced risk to schedule. If the AF bought a new design that required a new or immature 
production line, then what you state would be true. 


The cost of an airframe is based on the amortized business model developed. Generally speaking 
the value of an aircraft is higher in the beginning but not necessarily the price since “deals” are 
worked out to launch aircraft. Price usually goes up over time while value may decrease from a 
depreciation and marketplace perspective. Quality doesn’t change over time since the quality 
standard is a constant. Certainly there might be reliability growth issues. First of a model 
aircraft obviously take longer since you have to do the initial testing. Production is done at risk 
while testing is going on. The delivery is dependent on FAA certification and not on 
efficiencies. 

I would stress the three statements common in every manufacturing process. I agree the stated 
reasons, all conceptually within economy of scale, are concerns during early production. I offer 
the following addition information, production run aircraft are often between 70% to 90% of 
their maturation process. As the maturation process continues, less developmental support is 
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required allowing for staff reductions and additional process streamlining. 

2A. Agree: The aircraft price at the beginning of a production run is higher due to NRE, labor 
efficiencies (learning curve), product maturity risk, OEM pricing strategies, etc. In the 
commercial world, the configuration stays more constant than in our DOD environment, thus the 
commercial pricing structure remains fairly even through the production runs and savings from 
efficiencies are not passed on to the consumer unless product demand falls off, volume discounts 
apply, or configuration change/model change are eminent (closeout sale). 

2B. Disagree: In the FAA world, very little manufacturing process risk is undertaken in aircraft 
assembly. Technology maturity levels are very high for the most part so you won’t see the same 
uptick in quality issues as compared to an F-22 for example. 

2C. Agree: This is somewhat true about all manufacturers who have assembly issues in the 
early stages. Drivers here are likely due to differences in engineering estimates of assembly 
times, supplier deliver issues, equipment failures, etc. Production delay costs are more 
significant and painful for the commercial OEM’s as they do not have the government support of 
additional funding to cover cost overruns. In later stages, it is generally true that schedules are 
far more predictable and represent a lower risk to the government. 

The main risk we see purchasing up-front or for that matter at any time in a commercial 
environment is that the OEM owns the product until delivered - especially under a FAR 12 
acquisition where you are basically buying an end item. Once the contract is signed, the 
configuration can be changed (i.e. a product enhancement) without your knowledge or consent. 
We are dealing with a possible order this year for 15 Cessna 208 Caravans where the fleet might 
have 2 configurations based on a planned production model changeover in the middle of our 
order. For maintenance reasons, a mixed fleet drives cost for training, equipment, and part 
ordering/stocking 2 configurations. Engineering changes - which are commonplace in DOD 
OEM’s during a production run, are difficult or impossible to accomplish after the order has been 
placed. In almost every case, we end up performing post-delivery mods after receiving the green 
aircraft which adds cost due to disassembly/rework of the aircraft from the original assembly. 

The recently awarded KC-46 will be an exception to this model as Boeing has established a 
unique assembly line to incorporate Military products as the aircraft is being built negating the 
post production modification standard saving time and cost. 

2A. Partially Agree: The “Catalog Price” at the beginning of production is based on planned 
production costs, anticipated market demand and parametric comparisons of capabilities of like 
market equipment. When the consumer demand wanes, or if a customer is identified as 
‘preferred” by the OEM due to large future orders or standing preferred customer status, 
substantial cost reductions and/ or concessions are common. Although production efficiencies 
are realized as the production line and processes mature, these cost savings are seldom reflected 
in the “Catalog Price”. 

2B. Disagree: Manufacturing processes for commercial derivative aircraft are, for the most part, 
mature and routine. Unless the aircraft is being provided by a “Start Up” manufacturer, the risk is 
extremely low. It is doubtful that a startup manufacturer would make it through risk matrix or 
source selection process. The risk to the Government would be too great. 

2C. Partially Agree: As previously stated, If the OEM were a startup manufacturer, this might 
hold true. However since most commercial derivative aircraft are not start up’s, the production 
processes are already mature and in place. Material shortages may pose a slight risk, however 

this would be the exception and not the rule. _ 

Agree. 
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2A. Disagree. Buying a CDA that has been in production for a number of years should reduce 
the cost because all of the engineering and production line bugs have typically been resolved. 
While you always have continuous process improvement it doesn’t provide huge gains in 
efficiencies for a mature production line. Manufacturing pricing will typically be lower in the 
beginning because of a "bulk" buy upfront and competition. Also pricing may increase in later 
years because of diminishing parts availability from commercial vendors as the commercial fleet 
downsizes. 

2B. Disagree. For typical CDA production the production line is mature as the AF is procuring 
a proven CDA, not a new design. Most of the workforce is already experienced on the specific 
CDA line with proven training programs for new hires. While there is always room for 
improvement with any production line, the quick and easy "low hanging fruit" has already been 
picked when the line was new and immature. Process improvements to a mature line are harder 
to implement and likely do not carry the same dramatic levels of improvement. 

2C. Disagree. Typical CDA the AF procures are already being produced and mature. Thus 
reduced risk to schedule. If the AF bought a new design that required a new or immature 
production line, then what you state would be true. 


Question 3 Responses. 

Disagree. First, the AF would lose the intangible benefits of the FAA quality and safety 
oversight of the CDA weapon system. The FAA programs equate to the ISO 9000 and AS 9110 
standards that are recognized and accepted industry wide as the standard or "best practice". It 
gives the AF another layer of safety and quality oversight from the aviation experts (FAA) that 
cannot be achieved without them. Next the AF MUST determine what they want to do with the 
CDA upon retirement from the AF. Do they want to sell the aircraft to commercial operators or 
just salvage them? If the decision is to sell them to commercial operators, then FAA 
airworthiness certification really needs to be maintained throughout the life cycle of the aircraft. 
If not, the AF must review the maintenance records for the entire life of the aircraft and prove to 
the FAA that all maintenance, mods, repairs, parts, ADs/SBs, were accomplished correctly to the 
satisfaction of the FAA. This is a time consuming process and must be done for every tail 
number, not just for a type of aircraft. Will this apply to aircraft like the KC-46? That decision 
hasn’t been made yet. But it definitely applies to aircraft like the C-21, C-20, and T-43 that are 
reaching the end of their life cycle in the AF and are highly desirable to commercial operators. 
Lastly the fleet size really only impacts the availability of approved commercial parts. As the 
commercial fleet decreases in size the number of parts vendors decrease as well. But that’s not 
to say approved FAA parts will not be available. The AF can be certified by the FAA to produce 
a new part and repair them as well (either Part 121 or Part 145). This effort is not complex, just 
a "new" way of doing business. 

I disagree with the statement in that it implies that the only advantage of FAA certification of 
commercial derivative aircraft is in regard to sustainment. This devalues the potential benefits 
provided during the EMD phase of the program. Obtaining FAA certification of commercial 
derivative aircraft takes advantage of the commercial processes already in place and reduces the 
USAF manpower required to certify the aircraft by leveraging off of the FAA. AFI 62-601 states 
that FAA type certification is the preferred method of certifying airworthiness of USAF operated 
CDA. 
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There are many issues to work through for the military to reap the maximum benefits of CDA’s 
besides just having a large commercial fleet still in use. Commercial repair and parts pooling 
procedures are much different than Air Force procedures. Reparable parts in the commercial 
sector need to be tagged with an 8130-3 Airworthiness tag, the AF ALC’s presently cannot do 
this. Also, FAA traceability and segregation requirements for consumable parts are different. 
Yes. 

Disagree. Yes, it is true that there are more opportunities for savings with a robust commercial 
industry, but there are savings that will be realized through just maintaining FAA certification. 
Disagree. Your statement might be true if the Government was going to rely solely on the 
contractor for sustainment of the system acquired. Also, I'm not sure AFI 62-6 allows this 
flexibility. See my response to Questions 4 and 5. 

There seems to be a benefit if you are in an open parts pool, but in the case of a closed parts pool, 
I do not see the benefit of keeping the LIKE FAA certification. 

The benefit of FAA certification based on fleet size is a supply chain perspective. The real 
benefit is use of existing data and licensing rather than owning data. To recreate data for 
unlimited rites is a very expensive proposition since it will likely need to be reverse engineered. 
Maintaining commonality with the commercial variants allows for synergies and economies of 
scale that would not normally be available for a military aircraft. 

The size of the available fleet is important, but far more important is the ability to access the 
desired shared resources. The commercial parts pool and other scale issues are contractual 
arrangements where all participating partners have agreed to a specific obligations. The 
participating military force structure being supported by the common pool resources has not been 
accomplished to date. This collaboration has significant risk associated. It would be most 
embarrassing to have the parts pool partners exercise political options and keep one of our 
aircraft from completing its combat mission by withholding parts. This is conjecture, though 
throughout our own history the political exploits of withholding resources to achieve our 
objectives has been exercised. This is of course a two way street. 

Disagree. First, the AF would lose the intangible benefits of the FAA quality and safety 
oversight of the CDA weapon system. The FAA programs equate to the ISO 9000 and AS 9110 
standards that are recognized and accepted industry wide as the standard or "best practice". It 
gives the AF another layer of safety and quality oversight from the aviation experts (FAA) that 
cannot be achieved without them. 

Next the AF MUST determine what they want to do with the CDA upon retirement from the AF. 
Do they want to sell the aircraft to commercial operators or just salvage them? If the decision is 
to sell them to commercial operators, then FAA airworthiness certification really needs to be 
maintained throughout the life cycle of the aircraft. If not, the AF must review the maintenance 
records for the entire life of the aircraft and prove to the FAA that all maintenance, mods, 
repairs, parts, ADs/SBs, were accomplished correctly to the satisfaction of the FAA. This is a 
time consuming process and must be done for every tail number, not just for a type of aircraft. 
Will this apply to aircraft like the KC-46? That decision hasn’t been made yet. But it definitely 
applies to aircraft like the C-21, C-20, and T-43 that are reaching the end of their life cycle in the 
AF and are highly desirable to commercial operators. 

Lastly the fleet size really only impacts the availability of approved commercial parts. As the 
commercial fleet decreases in size the number of parts vendors decrease as well. But that’s not 
to say approved FAA parts will not be available. The AF can be certified by the FAA to produce 
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a new part and repair them as well (either Part 121 or Part 145). This effort is not complex, just 
a "new" way of doing business. 

Disagree: The FAA provides continuous monitoring of certification adherence well after delivery 
of the aircraft. Maintenance & repair processes are monitored regardless of the fleet size. If the 
equipment has an FAA certification, the FAA is mandated to provide oversight and the holder is 
required to remain in compliance. We would see benefits from a larger fleet size post-delivery in 
more repair centers around the country/world for service and a larger parts pool for spares. 
Disagree: The FAA provides continuous monitoring of certification adherence as well as 
enforcement of compliance with approved maintenance & repair processes regardless of the 
fleet size or type of aircraft. If the equipment has an FAA certification, the FAA is mandated to 
provide oversight and the holder is required to remain in compliance. 

Agree. A large commercial fleet enhances the ability for increased maintenance, supply and 
engineering support not only in the US but worldwide. 


Question 4 Responses. 

Number 1 is the total cost to field a new weapon system based on a military unique design (B-2) 
or a design based on a CDA (KC-46). The key is to restrict changes to a CDA design to a TRUE 
requirements for military specific missions...not just "neat to have" stuff. The MC-12 program is 
an outstanding example of how successful this can be! Also there is the potential to reduce the 
amount of spares bought up front due to "parts pooling" agreements the AF has with commercial 
carriers. This can potentially save BILLIONS of taxpayer dollars and definitely worth the work 
to establish these agreements. 

Much like automobiles, I feel that the number of aircraft produced and the number of different 
commercial operators of that aircraft have more of an influence on supply chain infrastructure, 
than when during the production run the aircraft are purchased. The larger the overall production 
run, the greater the availability of services and supplies. Case in point is the B737, first delivered 
in the late 1960's. With almost 7,000 aircraft delivered (>4500 still in service) and -2200 on 
order, and operated all over the world by over 500 airlines (8 operators within the USA alone) 
with a well-established supply and repair infrastructure that will be in place for years to come. 
There are still approximately 380 B737-200 (first model delivered of the B737 family) still in 
service. With such a large production run spanning over approx. 45 years, it is inherent that 
early adoption of the B737 provided benefits regarding the length of time that supplies and 
services would be available to those early operators, such as Southwest Airlines. Purchasing at 
the end of a planned production run can reduce risk in that it allows better insight into the size of 
the commercial supply chain and repair infrastructure to allow a better assessment as to the 
potential benefits available to the government purchaser. 

Air Force leaders must realize that there is benefit to using CDA’s on the developmental and 
production side, but differences in logistics, maintenance, and receiving inspection procedures 
between the Air Force and FAA standards, make many touted benefits on the sustainment side, 
such as parts pooling, difficult to achieve. 

Must consider AF goals as to whether to maintain organically or with contractor support. 
Commercial Derivative Aircraft (CDA) rely on a commercial parts pool and the necessary 
provisioning data to transition required parts into the AF supply system for full organic support 
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may not always be available. We are experiencing this very issue on the KC-46. In the long 
term when the AF becomes the only user of this aircraft it is imperative we have all necessary 
data to provision and possibly reverse engineer parts for repair when they no longer are available 
for procurement through commercial sources due to obsolescence. 

I think it is very beneficial to pursue utilizing commercial derivative aircraft. There are a lot of 
benefits such as COTS, supply chain management, large parts pool, and proven technology. 

I'm not sure where your next survey/questions will be directed, but I think it is very important 
additional infonnation is considered in each of the statements/questions above; we are not given 
the full context of the program that senior leaders will be making their decision in. For example, 
what is the long-term sustainment strategy after the CDA are acquired? If the government will 
be utilizing CLS for life, considering if the CDA is in the beginning/end of production may/may 
not be a primary consideration (contractor has to support the aircraft regardless). However, if 
after acquiring the CDA with the intent is to transition to a fully organic (government 
maintained) posture, I believe data rights (and cost) far exceed considering where the production 

cycle is. _ 

I think it is a good idea for savings. I just don't see the savings maintaining it as a commercial 

A/C. _ 

The real inefficiencies are driven by multiple contracts for the same thing rather than increasing 
the buying power of the DOD by combining requirements or using ID IQ contracts to standardize 
the business model used by the services. Buying the same things differently is very costly. 
Consideration should also be given to using commercial airlines to buy the basic aircraft and 
competing the modifications. That way you would just be a part of a larger airline purchase for 
the basic airframe and take better advantage of commercial business practices. 

Number 1 is the total cost to field a new weapon system based on a military unique design (B-2) 
or a design based on a CDA (KC-46). The key is to restrict changes to a CDA design to a TRUE 
requirements for military specific missions...not just "neat to have" stuff. The MC-12 program is 
an outstanding example of how successful this can be! Also there is the potential to reduce the 
amount of spares bought up front due to "parts pooling" agreements the AF has with commercial 
carriers. This can potentially save BILLIONS of taxpayer dollars and definitely worth the work 
to establish these agreements. 

The leadership recognized economies of scale have yet to be codified. There is significant risk 
with tying ourselves to an commercial aircraft for long tenn sustainment and support. In the 
forefront, there is the tie back to the manufacture for engineering support without recourse. This 
means unless the government provides significant investment, items previously captured in 
avoidance or savings, we are subject to the technology refresh cycles of industry. While 
partnering with industry will provide some small measure of influence, when it comes time 
produce a new aircraft industry will do it and phase out what may be our most important asset 
militarily. As an example, we see this process repeated every 3 to 5 years by Microsoft, other 
software manufacturer's, the auto industry and literally every other no commodity item. 

To summarize, industry operates with profit motivation. The aviation market is not within our 
control and we can only provide a limited stakeholder interaction. We are not the majority vote 
and will be influenced in ways we cannot for see at this time. 

See answer above to questions 1, 2 and 3. 

Here are few additional reasons for utilizing a commercial derivative aircraft; lower development 
cost, reduced testing schedule, quicker delivery to the field and lower upfront logistics cost by 
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tapping into existing commercial process and products. Utilization of commercial manufacturing 
and quality assurance oversight is also available. Safety of the aircraft is increased by using the 
existing civil airworthiness certificate already issued to the baseline aircraft. 


Question 5 Responses. 

I’ve attached a draft white paper that provides a comprehensive look at the issue of maintaining 
commercial derivative aircraft (CDA) airworthiness. The paper is authored by Mr. Jim Warren, 
a support contractor in our office supporting us in the area of airworthiness. Jim is an ASC/EN 
civilian retiree who has worked many CDA programs. Although the white paper title infers that 
it is regarding the KC-46, the infonnation applies to other CDA. Although the information in the 
paper is considered to be accurate, please treat the paper as a draft. I would recommend that you 
contact Jim at 51589 for the status of finalization of the paper. 

Pursue FAA certification or equivalency for the Air Force AFC’s, and ensure Air Force Supply 
chain policies and procedures can accommodate the management of FAA parts. 

FAA certification adds another layer of safety and quality oversight that should not be ignored. 
There have been deadly crashes (T-3, C-21, etc.) in CDA that would not have likely happened if 
the AF was truly committed to maintaining and operating these aircraft IAW FAA standards. 
FAA certification also aligns the AF with industry MRO leaders with a standard that is 
recognized worldwide. This opens doors to better communication/relationships with commercial 
providers, increased partnering possibilities with OEM’s, increased credibility within industry, 
enhanced mechanic certifications (A&P, etc.), and improved safety and quality programs. I’ve 
already mentioned how maintaining FAA airworthiness would eliminate the lengthy and costly 
audit for each tail number that the AF wants to sell to the public. Also the cost to certify the 76 
MXW FAA Part 145 repair station was less than $20,000...not the millions you hear mentioned 
by others. 

Must consider addressing 3 key process areas: Engineering, Maintenance, and Supply. 
Engineering- Must have manpower to support dual certifications (FAA and Military) Organic 
and Depot Fevel Maintenance-Must have FAA memo recognizing AF O-level maintenance as 
equivalent. Depots must obtain and sustain FAR Part 145 Certification. Retail/Wholesale 
Supply-Management and visibility of certified FAA parts—FAA certified parts require separate 
NSNs. Also must have control and constant visibility of FAA certified parts. AF data 
systems/processes not currently set up to manage FAA certified parts. 

Staying within the FAA system allows us to use all the authorized parts manufacturers/repair 
licensees, etc. as we see fit. If we were to not do that then they couldn’t use their licenses on 
anything for us and we would have to establish separate contracts and separate contract 
administration functions. There is also comfort in numbers. When analyzing the impact of 
design flaws, accidents, etc. making decisions on the basis of our data alone would be flawed. 
Then there is also the additional cost of doing anything unique or as a customer of one. A lot of 
the "savings" are ion reality cost avoidance which never gets recognized or analyzed. It all traces 
back to the design data and the approvals. 

I believe CDA can save the Government development funds, but as far as FAA certification we 
must fully understand the implications FAA certification has on the enterprise. My experience 
has been the Government may not be as well suited to retaining this certification and realizing 
the benefits expected. Commercial parts pools, data rights, commercial tech orders, FAR Part 
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145 certification, FAA "equivalency," all have proven challenging on KC-46. This doesn't even 
speak to parts traceability, accountability, IT system interfacing (G081/CEMS) that are all 
challenges yet to come. Recommend you devise a way that accounts for the moving parts in each 
program. The tool might be as simple as a decision tree that guides AOs or PMs in teeing up 
topics for decision makers through common considerations lumped together in categories. For 
example, to use your earlier case, FAA certification retention. What is the sustainment strategy 
(CLS, organic, PPP)? Are Core, 50/50 impacts expected? Are IT systems capable of supporting 
the desired approach? Is funding available to modify the IT systems if needed? Etc... 

I do not see the benefits of maintaining a FAA certified A/C with a closed parts pool. I can see 
the benefit of buying a FAA certified A/C for the cost savings, but not maintaining a like FAA 
certification. 

The USAF and DOD need to take a close look at the differences between the airline industry and 
their practices and align them better in order to take maximum advantage of the commercial 
business model. It has been proven to be more efficient and less costly. The USAF can no 
longer afford unique processes unless specifically justified. The “we have always done it this 
way” argument doesn’t hold water any longer. 

The comments for repair process and parts are no different than question 4. Once again, the risk 
is high we will be captured in the tech refresh cycles and be forced to recapitalize or replace 
usable aircraft due to industry influence. But this is an issue for tomorrow and it is unlikely the 
current decision makers will concern themselves past the a 15 year forecast. 

For design purposes, an FAA type certificate always benefits the Air Force for airworthiness 
reasons alone* (all of our aircraft are Military Certified for use), however the 50/50 rule for 
maintenance will always be a concern with determining who maintains the aircraft in a cost 
effective manner. Does it make sense to duplicate or compete with private industry for 
maintenance and/or repair of commercial aircraft to comply with the law - even for small fleet 
sizes? 

* Savings derived from not duplicating engineering analysis of design on the Military side 
Maintaining FAA certification is beneficial as well. I've listed a few of the advantages below. 
Configuration control. The FAA regulates this process and approves all changes. This keeps 
modifications to a standard configuration. FAA standards are recognized internationally. Safety 
operation of the aircraft and airworthiness standards issued by the FAA are accepted worldwide 
providing continuity for global operation. Use of parts from the commercial parts pool provides 
a larger source for replacement components without having to increase sparing levels. FAA 
database used to track part failures throughout the commercial fleet. This information is used to 
track failure statistics that may indicate changes in failure rates, repair vendor quality problems 
or possible design related issues. Safety of commercial derivative aircraft is enhanced by 
utilizing the commercial fleet’s access to technology and experience. Provides a greater base to 
access safety related topics that may surface within the commercial fleet. 

FAA certification adds another layer of safety and quality oversight that should not be ignored. 
There have been deadly crashes (T-3, C-21, etc.) in CDA that would not have likely happened if 
the AF was truly committed to maintaining and operating these aircraft IAW FAA standards. 
FAA certification also aligns the AF with industry MRO leaders with a standard that is 
recognized worldwide. This opens doors to better communication/relationships with commercial 
providers, increased partnering possibilities with OEM’s, increased credibility within industry, 
enhanced mechanic certifications (A&P, etc.), and improved safety and quality programs. I’ve 
already mentioned how maintaining FAA airworthiness would eliminate the lengthy and costly 
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audit for each tail number that the AF wants to sell to the public. Also the cost to certify the 76 
MXW FAA Part 145 repair station was less than $20,000...not the millions you hear mentioned 
by others. 
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Appendix D: Delphi Study Questionnaire One Revised Statements 


Revised Questionnaire Statements 

Two of the statements from the original questionnaire were modified based on inputs received. 
Modifications are italicized. 

Please review the following two statements below. Then, based on your current knowledge and 
experience, identify whether you agree or disagree. Also, please identify any items you would 
add or remove. 

1. Regarding Air Force purchase of commercial derivative aircraft, one of the risks associated 
with purchasing aircraft near the beginning of the planned production run is that the level of 
quality of aircraft purchased can be lower during initial stages of production when compared to 
later stages (due to employee experience gained over time, lessons learned, etc.) However, when 
well-established manufacturers and proven (mature) technology are used, quality is fairly 
constant throughout the production process and this risk is minimal. 

2. Many of the sustainment related benefits associated with maintaining or pursuing FAA 
certification of Air Force commercial derivative aircraft or repair processes can only be achieved 
when there is a large commercial fleet still in use by the commercial industry (benefits might 
include a larger parts pool or an increased number of repair centers, for example). However, 
regardless offeet size, additional benefits can be achieved through the use of existing FAA data, 
certifications, and processes. There are also benefits associated with additional FAA oversight of 
regulation compliance and maintenance practices. 
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Appendix E: Delphi Study Questionnaire Two 


Introduction: 

Based on responses received from participants in this study (as well as additional research), 
several key benefits to maintaining FAA certification on military commercial derivative aircraft 
(MCDA) were identified and are listed below (in no particular order). This list is not meant to be 
all-inclusive, but only to identify a majority of the most beneficial attributes associated with 
maintaining FAA certification (beneficial with regard to potential cost savings, improved safety, 
etc). The costs associated with maintaining FAA certification are not included in this portion of 
the study. The questions below should be answered based on your current level of knowledge 
and experience. 

Instructions: 

Please divide 100 points among the attributes listed below based on your estimation of their level 
of impact. For example, if you feel that being able to participate in the commercial parts pool for 
common commercial parts is the biggest benefit that comes from maintaining FAA certification 
of MCDA, then that attribute should receive the most points relative to the other attributes. You 
may also feel that some attributes provide no benefit at all, in which case you would assign zero 
points. Since these attributes were all identified as being potentially beneficial, no negative 
numbers should be used. Fractions are allowed, but the total of all points combined must 
equal 100, no more and no less. 


Attribute 

Score 

Participation in the commercial parts (spares) pool 


Access to established commercial maintenance support and infrastructure 


Access to an established commercial supply chain 


Use of existing manuals, repair/failure data, and technical data 


Use of existing FAA approved processes (Service Bulletin and Airworthiness 
Directive processes, repair processes, etc.) 


Improved control and tracking of aircraft configuration data 


Ability to participate in cross-servicing with European Aviation Safety 

Agency (EASA) and FAA fleets 


Ability to sell demilitarized MCDA back to commercial market 


Reduced need to acquire proprietary OEM design data or flight test data (this 
could also be considered the benefit associated with a reduced likelihood that 
“reverse engineering” would have to take place, assuming a continuing OEM 
relationship is maintained) 


Improved safety due to increased FAA oversight 


Total 

100 


If you feel the list is missing a substantial benefit, please try to account for it within one of the 
existing attributes (and annotate the table accordingly), or list it below along with the score you 
would assign (remember that your combined total must equal exactly 100). 




Instructions: 

Concerning the same potential benefits from the first part of the questionnaire, please consider how the 
impact of those benefits might change over the life of the fleet. Fill in the columns below by referencing 
the figures on the following page. Figures 1 through 9 on the following page represent varying 
behaviors with respect to time. The life cycle timeline at the top of the following page assigns numbers 
to various life cycle stages. An example is provided below. 

Example: 

If you feel that the benefit associated with participation in the commercial parts pool would be fairly 
constant once the fleet reached mid-production, but then reduce dramatically as the commercial fleet 
began to retire, you would choose Figure 4 as the most appropriate behavior graph and enter that figure 
number in the chart below. 

Next, utilizing the life cycle timeline at the top of the page, identify which numbers on the timeline 
correspond to points A and B in the figure. In this example, you would choose life cycle number 2 (mid 
production) for point A and lifecycle number 5 (commercial retirement) for point B, and enter those 
numbers in the appropriate columns below. If you’re unsure of which life cycle numbers to use, leave 
those columns blank. 


Attribute 

Figure 

(1-9) 

Life Cycle # 
for Point A 

Life Cycle # 
for Point B 

Participation in the commercial parts pool 

4 

2 

5 


Note: You only need to enter life cycle numbers for Figures 1-6, because Figures 7-9 do not contain 
points A and B. 


Fill in the columns below based on your best estimation of how these benefits might change over time. 


Attribute 

Figure 

(1-9) 

Life Cycle # 
for Point A 

Life Cycle # 
for Point B 

Participation in the commercial parts (spares) pool 




Access to established commercial maintenance support and 
infrastructure 




Access to an established commercial supply chain 




Use of existing manuals, repair/failure data, and technical 
data 




Use of existing FAA approved processes (Service Bulletin 
and Airworthiness Directive processes, repair processes, 
etc.) 




Improved control and tracking of aircraft configuration data 




Ability to participate in cross-servicing with European 
Aviation Safety Agency (EASA) and FAA fleets 




Ability to sell demilitarized MCDA back to commercial 
market 




Reduced need to acquire proprietary OEM design data or 
flight test data (this could also be considered the benefit 
associated with a reduced likelihood that “reverse 
engineering” would have to take place, assuming a 
continuing OEM relationship is maintained) 




Improved safety due to increased FAA oversight 
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EARLY 

PRODUCTION 


1 2 


MID 

PRODUCTION 


Fleet Life Cycle Timeline 

LATE COMMERCIAL 

PRODUCTION RETIREMENT 



4 


POST 

PRODUCTION 


5 


6 


MILITARY CDA 
RETIREMENT 


Behavior Over Time (Figures 1 through 9) 


FIGURE 1 


FIGURE 2 


FIGURE 3 



Increases dramatically from point A 
to point B and then levels off 



Increases from point A to a peak 
and then decreases to point B 



Fairly constant between points A 
and B, then increases dramatically 


FIGURE 4 


FIGURE 5 


FIGURE 6 





Fairly constant between points A 
and B, then decreases dramatically 


Decreases from point A to a low 
and then increases to point B 


Decreases dramatically from point 
A to point B and then levels off 


FIGURE 7 FIGURE 8 FIGURE 9 



Fairly constant rate of increase 
through all points 



Fairly constant rate through all 
points (little or no change) 



Fairly constant rate of decrease 
through all points 
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Append 

ix F. Attribute Weig 

its and Averages 


i 

2 

3 

4 

5 

6 

7 

8 

9 

10 

Total 

Mean 

Attribute 

Score 

Score 

Score 

Score 

Score 

Score 

Score 

Score 

Score 

Score 



Participation in the commercial parts 
(spares) pool 

15 

20 

35 

20 

20 

40 

20 

15 

20 

5 

210 

21.00 

Access to established commercial 
maintenance support and infrastructure 

10 

10 

10 

5 

10 

0 

2 

10 

5 

5 

67 

6.70 

Access to an established commercial 
supply chain 

10 

15 

20 

18 

20 

10 

12 

10 

20 

20 

155 

15.50 

Use of existing manuals, repair/failure 
data, and technical data 

10 

15 

10 

6 

10 

15 

9 

10 

10 

30 

125 

12.50 

Use of existing FAA approved 
processes (Service Bulletin and 
Airworthiness Directive processes, 
repair processes, etc.) 

15 

15 

5 

7 

5 

5 

8 

15 

9 

30 

114 

11.40 

Improved control and tracking of 
aircraft configuration data 

15 

10 

5 

12 

10 

0 

5 

5 

1 

1 

64 

6.40 

Ability to participate in cross-servicing 
with European Aviation Safety Agency 
(EASA) and FAA fleets 

10 

0 

1 

15 

5 

0 

3 

5 

2 

3 

44 

4.40 

Ability to sell demilitarized MC DA 
back to commercial market 

5 

5 

4 

9 

10 

25 

1 

5 

1 

1 

66 

6.60 

Reduced need to acquire proprietary 
OEM design data or flight test data 
(this could also be considered the 
benefit associated with a reduced 
likelihood that “reverse engineering” 
would have to take place, assuming a 
continuing OEM relationship is 
maintained) 

5 

5 

5 

8 

5 

0 

25 

0 

30 

0 

83 

8.30 

Improved safety due to increased 

FAA oversight 

5 

5 

5 

0 

5 

5 

15 

25 

2 

5 

72 

7.20 

Total 

100 

100 

100 

100 

100 

100 

100 

100 

100 

100 

1000 

100.00 
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Appendix G. Attribute Behavior Curves 



1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

. 

Figure 

Figure 

Figure 

Figure 

Figure 

Figure 

Figure 

Figure 

Figure 

Figure 


(1-9) 

(1-9) 

(1-9) 

(1-9) 

(1-9) 

(1-9) 

(1-9) 

(1-9) 

(1-9) 

(1-9) 

Participation in the commercial parts 
(spares) pool 

4 

4 

4 

6 

4 

4 

4 

1 

4 

4 

Access to established commercial 
maintenance support and infrastructure 

4 

4 

8 

6 

8 

4 

2 

2 

4 

5 

Access to an established commercial 
supply chain 

4 

4 

4 

4 

4 

4 

4 

3 

1 

4 

Use of existing manuals, repair/lailure 
data, and technical data 

6 

4 

8 

8 

8 

8 

1 

3 

8 

9 

Use of existing FAA approved processes 
(Service Bulletin and Airworthiness 
Directive processes, repair processes, 
etc.) 

8 

2 

4 

8 

8 

8 

1 

1 

8 

8 

Improved control and tracking of aircraft 
configuration data 

4 

2 

8 

6 

8 

8 

1 

7 

8 

9 

Ability to participate in cross-servicing 
with European Aviation Safety Agency 
(EASA) and FAA fleets 

6 

9 

8 

4 

8 


2 


8 

8 

Ability to sell demilitarized MC DA back 
to commercial market 

3 

3 

6 

6 

4 

4 

8 

2 

6 

8 

Reduced need to acquire proprietary 
OEM design data or flight test data (this 
could also be considered the benefit 

associated with a reduced likelihood that 
“reverse engineering” would have to take 
place, assuming a continuing OEM 
relationship is maintained) 

6 

3 

6 

4 

1 

9 

3 

4 

4 

3 

Improved safety due to increased FAA 
oversight 

8 

8 

8 

8 

8 

8 

1 

4 

8 

8 
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Appendix H. Attribute Behavior Curve Point A Identification 



1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

Attribute 

Life Cycle # 
for Point A 

Life Cycle # 
for Point A 

Life Cycle # 
for Point A 

Life Cycle # 
for Point A 

Life Cycle # 
for Point A 

Life Cycle # 
for Point A 

Life Cycle # 
for Point A 

Life Cycle # 
for Point A 

Life Cycle # 
for Point A 

Life Cycle # 
for Point A 

Participation in the commercial parts 
(spares) pool 

2 

1 

2 

1 

2 

2 

1 

4 

1 

2 

Access to established commercial 
maintenance support and infrastructure 

2 

4 


1 


2 

1 

4 

1 

1 

Access to an established commercial 
supply chain 

1 

1 

2 

2 

1 

2 

2 

2 

1 

1 

Use of existing manuals, repair/failure 
data, and technical data 

1 

1 





1 

1 



Use of existing FAA approved processes 
(Service Bulletin and Airworthiness 
Directive processes, repair processes, 
etc.) 


1 

1 




1 

4 



Improved control and tracking of aircraft 
configuration data 

1 

1 


1 



1 

1 



Ability to participate in cross-servicing 
with European Aviation Safety Agency 
(EASA) and FAA fleets 

1 



2 



2 




Ability to sell demilitarized MCDA back 
to commercial market 

1 

5 

4 

4 

3 

1 


5 

1 


Reduced need to acquire proprietary 
OEM design data or flight test data (this 
could also be considered the benefit 

associated with a reduced likelihood that 
“reverse engineering” would have to take 
place, assuming a continuing OEM 
relationship is maintained) 

1 

4 

1 

1 

1 


1 

1 

1 

1 

Improved safety due to increased FAA 
oversight 







1 

1 
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Appendix I. Attribute Behavior Curve Point B Identification 



1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

Attribute 

Life Cycle # 
for Point B 

Life Cycle # 
for Point B 

Life Cycle # 
for Point B 

Life Cycle # 
for Point B 

Life Cycle # 
for Point B 

Life Cycle # 
for Point B 

Life Cycle # 
for Point B 

Life Cycle # 
for Point B 

Life Cycle # 
for Point B 

Life Cycle # 
for Point B 

Participation in the commercial parts 
(spares) pool 

5 

5 

4 

5 

5 

5 

5 

5 

5 

5 

Access to established commercial 
maintenance support and infrastructure 

5 

5 


5 


5 

5 

5 

6 

6 

Access to an established commercial 
supply chain 

5 

5 

4 

5 

5 

5 

5 

5 

2 

5 

Use of existing manuals, repair/Mure 
data, and technical data 

4 

5 





2 

5 



Use of existing FAA approved processes 
(Service Bulletin and Airworthiness 
Directive processes, repair processes, 
etc.) 


6 

4 




4 

5 



Improved control and tracking of aircraft 
configuration data 

6 

5 


5 



6 

5 



Ability to participate in cross-servicing 
with European Aviation Safety Agency 
(EASA) and FAA fleets 

5 



5 



5 




Ability to sell demilitarized MCDA back 
to commercial market 

5 

6 

6 

5 

6 

5 


6 

5 


Reduced need to acquire proprietary 

OEM design data or flight test data (this 
could also be considered the benefit 

associated with a reduced likelihood that 
“reverse engineering” would have to take 
place, assuming a continuing OEM 
relationship is maintained) 

5 

5 

3 

4 

4 


1 

6 

6 

3 

Improved safety due to increased FAA 
oversight 







6 

6 
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Appendix J. Decision Support Tool 




MCDA Fleet Life Cycle Stai 

ges 



Early Production 

Mid Production 

Late 

Production 

Post 

Production 

Begin 

Commercial 

Retirement 

Begin MCDA 
Retirement 

FAA Certification 

Decision Criteria 
(Attributes) 

Attribute 

Weight 

Impact 

Level 

Impact 

Score 

Impact 

Level 

Impact 

Score 

Impact 

Level 

Impact 

Score 

Impact 

Level 

Impact 

Score 

Impact 

Level 

Impact 

Score 

Impact 

Level 

Impact 

Score 

Parts Pooling 

21.0 

0.99 

20.79 

0.98 

20.58 

0.95 

19.95 

0.86 

18.06 

0.63 

13.23 

0 

0.00 

Access to SC Infrastructure 

15.5 

0.99 

15.35 

0.98 

15.19 

0.95 

14.73 

0.86 

13.33 

0.63 

9.77 

0 

0.00 

Use of Existing Data 

12.5 

1 

12.50 

1 

12.50 

1 

12.50 

1 

12.50 

1 

12.50 

1 

12.50 

Use of FAA Processes 

11.4 

1 

11.40 

1 

11.40 

1 

11.40 

1 

11.40 

1 

11.40 

1 

11.40 

Access to Proprietary Data 

8.3 

1 

8.30 

0.99 

8.22 

0.98 

8.13 

0.95 

7.89 

0.86 

7.14 

0.63 

5.23 

Improved Safety/Quality 

7.2 

1 

7.20 

1 

7.20 

1 

7.20 

1 

7.20 

1 

7.20 

1 

7.20 

Maintenance Support 

6.7 

0.99 

6.63 

0.98 

6.57 

0.95 

6.37 

0.86 

5.76 

0.63 

4.22 

0 

0.00 

Commercial Sell Back 

6.6 

0 

0.00 

0 

0.00 

0 

0.00 

1 

6.60 

0.37 

2.44 

0.14 

0.92 

Improved Config Control 

6.4 

1 

6.40 

1 

6.40 

1 

6.40 

1 

6.40 

1 

6.40 

1 

6.40 

FAA/EASA Cross Servicing 

4.4 

1 

4.40 

1 

4.40 

1 

4.40 

1 

4.40 

1 

4.40 

1 

4.40 

Totals 

100 ^ 


92.97 

92.45 

91.07 

93.54 

78.70 

48.05 
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Appendix K. Decision Support Tool Output 


Overall Attribute Impact Level Over Time 



Early Production Mid Production Late Production Post Production Begin Commercial Begin MCDA 

Retirement Retirement 


■ Relative Impact 


Lifecycle Timeline 
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Appendix L. Attribute Comparison 


Specific Attribute Impact Level Over Time 


25.00 


20.00 


Si 15.00 


£ 10.00 


5.00 


0.00 



Parts Pooling 


Access to SC 
Infrastructure 


Use of Existing Data Use of FAA Processes 


Specific Attribute 


Improved 

Safety/Quality 


*Only those attributes 
with 50% or more Delphi 
concurrence on behavior 
are shown 


■ Early Production 

■ Mid Production 

■ Late Production 

■ Post Production 

Begin Commercial Retirement 
Begin MCDA Retirement 
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Appendix M. Attribute Comparison (Sustainment) 


Specific Attribute Impact Level Over Time (Sustainment) 


30.00 


25.00 



0.00 


Parts Pooling Use of Existing Data Access to SC Access to Proprietary Commercial Sell Back 

Infrastructure Data 


Specific Attribute 


*Top five attributes by 
weight, as assigned by 
participants in 
sustainment related fields 


■ Early Production 

■ Mid Production 

■ Late Production 

■ Post Production 

Begin Commercial Retirement 
Begin MCDA Retirement 
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Appendix N. Attribute Comparison (Acquisition) 


Specific Attribute Impact Level Over Time (Acquisition) 


30.00 


25.00 


20.00 



PartsPooling Access to SC Use of Existing Data Use of FAA Processes Maintenance Support 

Infrastructure 

Specific Attribute 


*Top five attributes by 
weight, as assigned by 
participants in acquisition 
related fields 


■ Early Production 

■ Mid Production 

■ Late Production 

■ Post Production 

Begin Commercial Retirement 
Begin MCDA Retirement 
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Appendix O: Research Request Memo 



DEPARTMENT OF THE AIR FORCE 

AIR FORCE MATERIEL COMMAND (AFMC) 


0 7 OCT 2011 


MEMORANDUM FOR ASC/WKI., OC-ALC/GKA, AMC/A4M, AMC/A5Q 

FROM: HQ AFMC/A4D 

4375 CHIDLAW RD 
WPAFB, OH 45433 

SUBJECT: Request for Commercial Derivative Aircraft Research Project Input 

1. As the designated representative for the HQ AFMC/A4 for studies, 1 am requesting your 
support with one of our sponsored research projects to identify ways to reduce the costs of new 
aircraft acquisition and sustainment. In particular, this project seeks to quantify the benefits and 
costs associated with Air Force use of FAA certified commercial derivative aircraft (CDA) and 
to develop a tool to assist with future decisions. This research can only be successful through 
your active participation and we will be pleased to share all of our results with you. 

2. Wc ask that you identify five individuals from your organization to review a set of decision 
criteria regarding Air Force use of CDA and FAA certification processes. Individuals selected to 
participate must be military or DoD civilian personnel and should be familiar with Air Force 
acquisitions of commercial derivative aircraft (i.e. C-27J, KC-46A, etc.). After reviewing a few 
paragraphs of material, participants will be asked to answer five questions based on their 
knowledge and experience. Subsequent survey rounds (a maximum of four) may be required to 
clarify responses and/or reach agreement amongst participants. Estimated time to complete the 
initial review and answer questions should not exceed 30 minutes. Participant anonymity will be 
preserved in all project reports. 

3. The research project point of contact is Capt Mike Low - Phone 520-481 -9246; E-mail - 
Michael.low@afit.edu. 



Chief, Depot Operations Division 
Directorate of I .ogistics 
Wright-Patterson AFB, OH 


Attachment: 

1. Questionnaire with Instructions 
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Appendix P. Human Experimentation Requirements Exemption Letter 



DEPARTMENT OF THE AIR FORCE 

AIR FORCE INSTITUTE OF TECHNOLOGY 
WRIGHT-PATTERSON AIR FORCE BASE OHIO 


29 September 2011 


MEMORANDUM FOR DR. ALAN JOHNSON 

FROM: William A. Cunningham. Ph D. 

AFTT IRB Research Reviewer 
2950 Hobson Way 

Wright-Patterson AFB. OH 45433-7765 

SUBJECT: Approval for exemption request from human experimentation requirements (32 C'FR 
219, DoDD 3216.2 and AFI40-402) for Delphi Study of Commercial Derivative Aircraft and 
Certification Criteria. 

1. Your request was based on the Code of Federal Regulations, title 32, part 219, section 101, 
paragraph (b) (2) Research activities that involve the use of educational tests (cognitive, 
diagnostic, aptitude, achievement), survey procedures, interview procedures, or observation of 
public behavior unless: (i) Information obtamed is recorded m such a manner that human 
subjects can be identified, directly or through identifiers linked to the subjects; and (ii) Any 
disclosure of the human subjects’ responses outside the research could reasonably place the 
subjects at risk of criminal or civil liability or be damaging to the subjects’ financial standing, 
employability, or reputation. 

2. Your study qualifies for this exemption because you are not collecting sensitive data, which 
could reasonably damage the subjects’ financial standing, employability, or reputation. Further, 
the demographic data you are collecting cannot realistically be expected to map a given response 
to a specific subject. 

3. This determination pertains only to the Federal, Department of Defense, and Air Force 
regulations that govern the use of human subjects in research. Further, if a subject’s future 
response reasonably places them at risk of criminal or civil liability or is damaging to their 
financial standing, employability, or reputation, you are required to file an adverse event report 
with this office immediately. 


WILLIAM A. CUNNINGHAM. PH D. 
AFIT Research Reviewer 
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Appendix Q. Delphi Study Participant Pool 


Aeronautical Systems Center (AFMC) 
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Appendix R: Quad Chart 



Impact of Decision Criteria on Federal 
Aviation Administration Certification of 
Military Commercial Derivative Aircraft 


TT?’T r r 


INTRODUCTION 


The decision regarding whether to maintain Federal Aviation 
Administration certification of military commercial derivative aircraft 
(MCDA) can be extremely difficult due to the subjective nature of some of 
the associated costs and benefits. The purpose of this research is to help 
decision makers understand the level of impact those costs and benefits have 
on their decision and how that impact may change over the lifecycle of the 
MCDA fleet. Data regarding the various criteria involved m such decisions 
were collected from subject matter experts using the Delphi technique. This 
data were then incorporated mto a decision support tool that uses the 
principles of multi-attribute decision making. The tool helps to illustrate the 
relative importance of each of the decision criteria based upon specific 
points m a typical MCDA fleet lifecycle. 


RESEARC H QUESTIONS 


What are the decision criteria USAF leaders should consider when 
determining whether or not to maintain FAA certification of MCDA? 
How do the costs and benefits associated with maintaining FAA 
certification of MCDA change over time? 

If the choice is made to maintain FAA certification of MCDA, how long 
should it be maintained ? 


DEC ISION CRITERIA 


ASSUMPTIONS 


Decision to utilize MCDA has already been made by USAF 
Most costs associated with FAA certification are sunk or fixed costs 
Attribute behavior can be described using exponential and linear curves 
Military benefits associated with maintaining FAA certification are 
allowed by commercial industry (i.e. access to commercial parts pool) 


Capt Mike Low 
Advisor: Dr. Alan Johnson 

Department of Operational Sciences (ENS) 
Air Force Institute of Technology 


MULTI-ATTRIBUTE DECISION SUPPORT TOOL 


1. Participation m the commercial parts (spares) pool 

2. Access to established commercial mamtenance support and infrastructure 

3. Access to an established commercial supply chain 

4. Use of existing manuals, repair/failure data, and technical data 

5. Use of existing FAA approved processes (Service Bulletin and 
Airworthiness Directive processes, repair processes, etc.) 

6. Improved control and tracking of aircraft configuration data 

7. Ability to participate in cross-servicing with European Aviation Safety 
Agency (EASA) and FAA fleets 

8. Ability to sell demilitarized MCDA back to commercial market 

9. Reduced need to acquire proprietary OEM design data or flight test data 
10.Improved safety due to mcreased FAA oversight 


Fleet Lifecycle Timeline 


Behavior-Over-Time Curves 




Lifecycle Impact Scores 


Sponsor 
AFMC/A4D 
Mr. Mike Dawson 



MCDA Fleet Life Cycle Sta 

jes 



Eanyr Production 

Mid Product on 

U 

Prod 

le 

cton 

P 

Prod 

Mt 

icd on 

Be 

Rstir 

1*" 

Begin 

Fisor 

" C .m 

FAA Certifies 13on 

Decision Criteria 
(Attributes) 

Weight 

Impact 


Impact 


tit pact 

iT.pact 

ITpac' 

tsr 

Impact 

E? 

Impact 



21.0 

0 35 


C 53 


1 ■ 


1 K 


■ a 




Access to SC mrasttucture 

15.5 

0.99 


0.98 


0 95 


0 86 


0.63 




use o' Esstlra Data 

12.5 


12.50 


12.50 








12.50 


11.4 


11.40 

1 

11 40 

1 


1 


1 


1 

It 40 

Access to =rocoetar> Data 

8 3 

1 

£ j: 

0.99 

822 

0 98 


095 


: 86 


0.63 

5.23 












































6.4 













FAXEASA Cross Servlet*) 

4.4 













Tome 100 

92.97 

92.45 

91.07 

93 54 

78.70 

48.05 



Overall Attribute Impact Level Over Time 

Mill 


I 


Specific Attribute Impact level Over Tl 



ll 


II 


RESULTS 


Parts pooLsupply chain access and use of data represent half of total impact 
Parts pool and supply chain access benefits diminish rapidly at commercial 
fleet retirement, but use of data benefit remains steady over timeline 
Remaining attribute behavior is mixed, but most remain steady over tune 
While overall attribute impact diminishes as commercial fleet retires, much 
of the benefit to maintaining FAA certification remains to end of lifecycle 


FUTURE RESEARCH 


Accurately identify costs associated with upgrading military systems to 
allow unlimited access to commercial supply cham parts and infrastructure 
Expand topic to DoD level and include all branches in the Delphi study 
Consider streamlining of military and FAA certification processes 
Consider FAA certification as a step toward international certification 


Department of Operational Sciences 
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